v23i040: Flex, a fast lex replacement, Part04/10
Rich Salz
rsalz at bbn.com
Fri Oct 12 01:28:24 AEST 1990
Submitted-by: Vern Paxson <vern at cs.cornell.edu>
Posting-number: Volume 23, Issue 40
Archive-name: flex2.3/part04
#! /bin/sh
# This is a shell archive. Remove anything before this line, then feed it
# into a shell via "sh file" or similar. To overwrite existing files,
# type "sh file -c".
# The tool that generated this appeared in the comp.sources.unix newsgroup;
# send mail to comp-sources-unix at uunet.uu.net if you want that tool.
# Contents: MISC/Atari.patches flex.1
# Wrapped by rsalz at litchi.bbn.com on Wed Oct 10 13:24:00 1990
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
echo If this archive is complete, you will see the following message:
echo ' "shar: End of archive 4 (of 10)."'
if test -f 'MISC/Atari.patches' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'MISC/Atari.patches'\"
else
echo shar: Extracting \"'MISC/Atari.patches'\" \(29474 characters\)
sed "s/^X//" >'MISC/Atari.patches' <<'END_OF_FILE'
X(Message inbox:32)
XDate: Mon, 03 Jul 89 21:15:32 CET
XFrom: V61%DHDURZ1.BITNET at lbl.gov
XSubject: Flex, bug fix, improvments, patches for Minix & TOS
XTo: vern at lbl-csam.arpa
X
XAt first I have to thank you for your wonderful program. I had ported the
Xold version to OS9,TOS (Atari ST) and Minix and the new version 2.1 Beta
Xto Minix and TOS.
X
XWhile porting and using flex I detected a bug and made some improvements.
XI have included a shared, compressed and uuencoded file contaning all cdiffs
Xand additional files (Sorry, but I'm on EBCDIC-Bitnet) and a short discussion
Xof the changes. Even some of the TOS specific changes might be of general
Xinterest !
X
XI posted these cdiffs to the minix discussion group, but I think it's up
Xto you to post them to the unix-sources group. If you plan to post even
Xthe TOS compiler specific patches please contact me because there might be
Xfurther compiler (P.D.) additions. If you have an interest I could also
Xport the new version to OS9 -- this is a little bit more difficult, because
XOS9 uses CR as end of line character (the EOL char. is coded into the
Xinitscan.c tables,...). It is necessary to change all occurences of '\n' to
Xmacros and variables and it's useful to add a new -n options (see commented
Xline in main.c)
X
X
X
XThe changes: (1.7.89 RAL)
X
X - Bug fix: The original flex didn't like trailing spaces in exclusive start
X condition lists ! If you add an trailing space to line 68 in scan.l
X
X "%x SECT2 SECT2PROLOG SECT3 CODEBLOCK PICKUPDEF SC CARETISBOL NUM QUOTE "
X
X you get a misleading error message:
X
X "Syntax error at line 69: bad start condition list"
X
X This bug can either be fixed in parse.y or in scan.l . I have chosen the
X last because there the fix is minimal: Just change the rule (line 128)
X
X "\n" to "{OPTWS}\n"
X
X - Enhancements:
X - new option "-?" that provides some help information about the other
X flags (main.c)
X - new option "-aTMPPATH" that allows a redefinition of the standard
X path for the temporary file (this might be rather large >200k if
X F,f options are selected). (main.c, flexdef.h (l.376))
X - hexdump of illegal characters -- this proved to be a useful debugging
X tool especialy if invisible control characters occur which weren't
X covered by the rules. (scan.l fprintf statement line 129,...)
X
X - Patches due to TOS
X - General: TOS uses CR,LF as end of line char., Flex wants only a single
X LF as EOL char. Therefore all I/O must be translated using f* calls.
X This is done everywhere besides the YY_INPUT macro (flex.skel (scan.c),
X line 31) that uses a low level 'read'. This should be definitly changed
X to fread, so that all I/O calls occur on the same level.
X - the short action_file_name has been "flexXXXXXX.tmp", but that's too
X much for TOS,MSDOS ! I changed it to "flexXXXX.tmp" in main.c (patch
X covered by the -a option additions)
X - some compilers don't like external names that are ambiguous within
X the first 8 characters. I defined macros that change all these long
X external names to names that are unique within the first 8 characters.
X Just define SHORT_EXTERNAL_NAMES to use this feature (flexdef.h)
X - some statement changes that some compiler don't like:
X typedef *xxx[] -> typedef **xxx (flexdef.h.,l.308)
X "/*" -> '/','*' within a comment in (scan.l, l.209)
X - changed short "lexyy.c" to "lex_yy.c" what's more similar to the unix
X "lex.yy.c" (main.c).
X - a few further really compiler dependent changes provided with
X #ifdef ATARI && LATTICE res. TURBO braces.
X
X - Additional TOS only files
X - Makefile.tos: Common makefile for all TOS compilers. If you add further
X ones please email me the new makefile.
X - flex.lnk: Lattice - GST linker argument extension file
X - flex.tlk: Turbo linker argument extension file
X
X
XAdditional remarks:
X
XI didn't add a new initscan.c (= flex -ist scan.l). The current one is good
Xenough for a first compilation. With this first version of flex you can
Xrebuild your own scan.c and the final flex version !
X
XMinix ST :
X - I had to "chmem =70000 cv" (>50000) to get flex linked
X - 'memset' (PC 1.3, EFTH40,...) is necessary
X - chmem =90000 flex may be sufficient
X
XMinix PC :
X It should be possible to port Flex to Minix PC. The current sizes of flex
X are:
X Minix ST (ACK) Lattice (TOS) Turbo (TOS)
X
X size 75300 83305 57957
X compilation time 22' 15' 3'40"
X flex -is scan.l 1'49" 43" 30"
X
X The Minix ST size includes the bad generated code using only a subset of
X the 68000 commands, long addresses only and a huge relocation table.
X Therefore the PC size will be <64 k ! More serious is the fact that I had
X to chmem =90000 flex to get scan.l converted to scan.c . But I never saw
X a more complex lex source than scan.l -- so it should be possible to
X reduce some array sizes without limitation for all day usage.
X
X No one volunteered yet for a Minix PC port -- but if someone will try it
X I would provide him with a new scan.c and some hints.
X
XTOS:
X Don't forget to adapt the flexskel path within flexdef.h !
X
X
XBitnet: V61 at DHDURZ1 Ronald Lamprecht
XUUCP: ...!unido!DHDURZ1.bitnet!V61 Theoretische Physik
XARPAnet: V61%DHDURZ1.BITNET at CUNYVM.CUNY.EDU (Heidelberg, West Germany)
X(Message inbox:36)
XDate: Wed, 05 Jul 89 21:16:07 CET
XFrom: V61%DHDURZ1.BITNET at csa2.lbl.gov
XSubject: Re: What is TOS
XTo: vern at lbl-csam.arpa
X
XTOS is the name of the Atari ST operating system that is very similar
Xto MSDOS (Both use CR,LF as end of line character). Therefore I have
Xbeen astonished that no EOL convertion porblems occur within MSDOS.
X
XI have been aware of the double buffering when changing read to fread and I
Xaccept your argument of a possible slow down. But if you integrate the other
XAtari - TOS changes, please insert a
X #ifdef ATARI
X fread ....
X #else
X read ....
X #endif
Xin flex.skel .
X
XBitnet: V61 at DHDURZ1 Ronald Lamprecht
XUUCP: ...!unido!DHDURZ1.bitnet!V61 Theoretische Physik
XARPAnet: V61%DHDURZ1.BITNET at CUNYVM.CUNY.EDU (Heidelberg, West Germany)
X
X
X
X
Xecho x - Makefile_cdiff
Xsed '/^X/s///' > Makefile_cdiff << '/'
XX*** Src_2.1/Makefile Thu Jun 28 00:06:42 1989
XX--- Makefile Thu Jul 3 02:12:48 1989
XX***************
XX*** 5,10 ****
XX--- 5,11 ----
XX # Porting considerations:
XX #
XX # For System V Unix machines, add -DSYS_V to CFLAGS.
XX+ # For Minix (ST), add -DSYS_V to CFLAGS
XX # For Vax/VMS, add -DSYS_V to CFLAGS.
XX # For MS-DOS, add "-DMS_DOS -DSYS_V" to CFLAGS. Create \tmp if not present.
XX # You will also want to rename flex.skel to something with a three
XX***************
XX*** 21,28 ****
XX SKELETON_DIR = /usr/local/lib
XX SKELETON_FILE = flex.skel
XX SKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
XX! CFLAGS = -O
XX! LDFLAGS = -s
XX
XX FLEX_FLAGS =
XX FLEX = ./flex
XX--- 22,29 ----
XX SKELETON_DIR = /usr/local/lib
XX SKELETON_FILE = flex.skel
XX SKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
XX! CFLAGS = -O -DSYS_V
XX! LDFLAGS =
XX
XX FLEX_FLAGS =
XX FLEX = ./flex
XX***************
XX*** 57,63 ****
XX yylex.c
XX
XX flex : $(FLEXOBJS)
XX! $(CC) $(CFLAGS) -o flex $(LDFLAGS) $(FLEXOBJS)
XX
XX first_flex:
XX cp initscan.c scan.c
XX--- 58,65 ----
XX yylex.c
XX
XX flex : $(FLEXOBJS)
XX! $(CC) $(CFLAGS) -o flex $(FLEXOBJS) $(LDFLAGS)
XX! chmem =150000 flex
XX
XX first_flex:
XX cp initscan.c scan.c
X/
Xecho x - flex.skel_cdif
Xsed '/^X/s///' > flex.skel_cdif << '/'
XX*** Src_2.1/flex.skel Thu Jun 28 00:19:20 1989
XX--- flex.skel Thu Jul 2 22:18:31 1989
XX***************
XX*** 28,34 ****
XX * is returned in "result".
XX */
XX #define YY_INPUT(buf,result,max_size) \
XX! if ( (result = read( fileno(yyin), buf, max_size )) < 0 ) \
XX YY_FATAL_ERROR( "read() in flex scanner failed" );
XX #define YY_NULL 0
XX #define yyterminate() return ( YY_NULL )
XX--- 28,34 ----
XX * is returned in "result".
XX */
XX #define YY_INPUT(buf,result,max_size) \
XX! if ( (result = fread( buf,1, max_size, yyin )) < 0 ) \
XX YY_FATAL_ERROR( "read() in flex scanner failed" );
XX #define YY_NULL 0
XX #define yyterminate() return ( YY_NULL )
X/
Xecho x - flexdef.h_cdif
Xsed '/^X/s///' > flexdef.h_cdif << '/'
XX*** Src_2.1/flexdef.h Thu Jun 28 00:43:27 1989
XX--- flexdef.h Thu Jul 3 02:45:50 1989
XX***************
XX*** 26,31 ****
XX--- 26,40 ----
XX
XX /* @(#) $Header: flexdef.h,v 2.0 89/06/20 15:49:50 vern Locked $ (LBL) */
XX
XX+ #ifdef ATARI
XX+ #define SYS_V
XX+ #define abs(x) ((x) < 0 ? -(x) : (x))
XX+ #define SHORT_FILE_NAMES
XX+ #ifdef TURBO
XX+ #define SHORT_EXTERNAL_NAMES
XX+ #endif
XX+ #endif
XX+
XX #ifndef FILE
XX #include <stdio.h>
XX #endif
XX***************
XX*** 41,47 ****
XX #endif
XX
XX #ifndef VMS
XX! char *memset();
XX #else
XX /* memset is needed for old versions of the VMS C runtime library */
XX #define memset(s, c, n) \
XX--- 50,58 ----
XX #endif
XX
XX #ifndef VMS
XX! #ifndef ATARI && TURBO
XX! char *memset();
XX! #endif
XX #else
XX /* memset is needed for old versions of the VMS C runtime library */
XX #define memset(s, c, n) \
XX***************
XX*** 81,91 ****
XX--- 92,129 ----
XX #define true 1
XX #define false 0
XX
XX+ #ifdef ATARI
XX+ #define DEFAULT_SKELETON_FILE "D:\\include\\flexskel"
XX+ #endif
XX+
XX
XX #ifndef DEFAULT_SKELETON_FILE
XX #define DEFAULT_SKELETON_FILE "flex.skel"
XX #endif
XX
XX+ #ifdef SHORT_EXTERNAL_NAMES
XX+ /* avoid long external names that are ambiguous within the first 8 characters */
XX+ #define current_mns c__mns
XX+ #define current_max_rules c__max_rules
XX+ #define current_max_state_type c__max_state_type
XX+ #define current_max_scs c__max_scs
XX+ #define current_max_dfa_size c__max__size
XX+ #define current_max_xpairs c__max_xpairs
XX+ #define current_max_template_xpairs c__max_template_xpairs
XX+ #define current_max_dfas c__max_dfas
XX+ #define current_maxccls c__maxccles
XX+ #define current_max_ccl_tbl_size c__max_ccl_tbl_size
XX+ #define indent_puts ind_puts
XX+ #define indent_put2s ind_put2s
XX+ #define gen_next_compressed_state gen_n_comressed_state
XX+ #define gen_next_match gen_n_match
XX+ #define gen_next_state gen_n_state
XX+ #define variable_trailing_context_rules var_tr_context_rules
XX+ #define variable_trailing_rule var_tr_rule
XX+ #define backtrack_report backtr_report
XX+ #define backtrack_file backtr_file
XX+ #endif
XX+
XX /* special chk[] values marking the slots taking by end-of-buffer and action
XX * numbers
XX */
XX***************
XX*** 305,311 ****
XX int int_val;
XX } ;
XX
XX! typedef struct hash_entry *hash_table[];
XX
XX #define NAME_TABLE_HASH_SIZE 101
XX #define START_COND_HASH_SIZE 101
XX--- 343,349 ----
XX int int_val;
XX } ;
XX
XX! typedef struct hash_entry **hash_table;
XX
XX #define NAME_TABLE_HASH_SIZE 101
XX #define START_COND_HASH_SIZE 101
XX***************
XX*** 372,378 ****
XX extern int datapos, dataline, linenum;
XX extern FILE *skelfile, *yyin, *temp_action_file, *backtrack_file;
XX extern char *infilename;
XX! extern char action_file_name[];
XX
XX
XX /* variables for stack of states having only one out-transition:
XX--- 410,416 ----
XX extern int datapos, dataline, linenum;
XX extern FILE *skelfile, *yyin, *temp_action_file, *backtrack_file;
XX extern char *infilename;
XX! extern char *action_file_name;
XX
XX
XX /* variables for stack of states having only one out-transition:
X/
Xecho x - main.c_cdiff
Xsed '/^X/s///' > main.c_cdiff << '/'
XX*** Src_2.1/main.c Thu Jun 28 00:30:39 1989
XX--- main.c Thu Jul 3 02:27:47 1989
XX***************
XX*** 81,96 ****
XX FILE *temp_action_file;
XX FILE *backtrack_file;
XX int end_of_buffer_state;
XX! #ifndef SHORT_FILE_NAMES
XX! char action_file_name[] = "/tmp/flexXXXXXX";
XX! #else
XX! char action_file_name[] = "flexXXXXXX.tmp";
XX! #endif
XX!
XX #ifndef SHORT_FILE_NAMES
XX static char outfile[] = "lex.yy.c";
XX #else
XX! static char outfile[] = "lexyy.c";
XX #endif
XX static int outfile_created = 0;
XX
XX--- 81,91 ----
XX FILE *temp_action_file;
XX FILE *backtrack_file;
XX int end_of_buffer_state;
XX! char *action_file_name;
XX #ifndef SHORT_FILE_NAMES
XX static char outfile[] = "lex.yy.c";
XX #else
XX! static char outfile[] = "lex_yy.c";
XX #endif
XX static int outfile_created = 0;
XX
XX***************
XX*** 328,333 ****
XX--- 323,329 ----
XX {
XX int i, sawcmpflag, use_stdout;
XX char *arg, *skelname = NULL, *flex_gettime(), clower(), *mktemp();
XX+ char *tmp_action =(char *)0, *malloc();
XX
XX printstats = syntaxerror = trace = spprdflt = interactive = caseins = false;
XX backtrack_report = performance_report = ddebug = fulltbl = fullspd = false;
XX***************
XX*** 349,354 ****
XX--- 345,355 ----
XX for ( i = 1; arg[i] != '\0'; ++i )
XX switch ( arg[i] )
XX {
XX+ case 'a':
XX+ if ( i != 1 )
XX+ flexerror( "-a flag must be given separately" );
XX+ tmp_action = &arg[i+1];
XX+ goto get_next_arg;
XX case 'b':
XX backtrack_report = true;
XX break;
XX***************
XX*** 445,452 ****
XX printstats = true;
XX break;
XX
XX! default:
XX! lerrif( "unknown flag %c", (int) arg[i] );
XX break;
XX }
XX
XX--- 446,458 ----
XX printstats = true;
XX break;
XX
XX! case '?':
XX! flexinfo(0);
XX! break;
XX!
XX! default:
XX! fprintf(stderr,"flex : unknown flag %c\n", (int) arg[i] );
XX! flexinfo(1);
XX break;
XX }
XX
XX***************
XX*** 454,459 ****
XX--- 460,493 ----
XX ;
XX }
XX
XX+
XX+ /* if you change the default tmp file names don't forget to change the
XX+ initialization for i, too !
XX+
XX+ */
XX+ #ifndef SHORT_FILE_NAMES
XX+ i = 10;
XX+ if (!tmp_action) i += 5;
XX+ #else
XX+ i = 12;
XX+ #endif
XX+ if (tmp_action)
XX+ i += strlen(tmp_action) + 1;
XX+ if((action_file_name = malloc(i+1)) == (char *)0)
XX+ flexerror("No memory for action_file_name");
XX+ *action_file_name = (char) NULL;
XX+ if (tmp_action)
XX+ strcat(action_file_name,tmp_action);
XX+ #ifndef SHORT_FILE_NAMES
XX+ else
XX+ strcat(action_file_name,"/tmp");
XX+ strcat(action_file_name,"/flexXXXXXX");
XX+ #else
XX+ if (tmp_action)
XX+ strcat(action_file_name,"/");
XX+ strcat(action_file_name,"flexXXXX.tmp");
XX+ #endif
XX+
XX if ( (fulltbl || fullspd) && usemecs )
XX flexerror( "full table and -cm don't make sense together" );
XX
XX***************
XX*** 520,526 ****
XX if ( (skelfile = fopen( skelname, "r" )) == NULL )
XX lerrsf( "can't open skeleton file %s", skelname );
XX
XX! (void) mktemp( action_file_name );
XX
XX if ( (temp_action_file = fopen( action_file_name, "w" )) == NULL )
XX lerrsf( "can't open temporary action file %s", action_file_name );
XX--- 554,562 ----
XX if ( (skelfile = fopen( skelname, "r" )) == NULL )
XX lerrsf( "can't open skeleton file %s", skelname );
XX
XX! #ifndef ATARI && LATTICE
XX! (void) mktemp( action_file_name );
XX! #endif
XX
XX if ( (temp_action_file = fopen( action_file_name, "w" )) == NULL )
XX lerrsf( "can't open temporary action file %s", action_file_name );
XX***************
XX*** 566,571 ****
XX--- 602,640 ----
XX }
XX
XX
XX+ flexinfo(status)
XX+ int status;
XX+ {
XX+ fprintf(stderr,"Syntax : FLEX inp_file\n");
XX+ fprintf(stderr,"Function: fast lexical analyzer generator V%s\n",flex_version);
XX+ fprintf(stderr,"Options : a dir_path : directory path for temporary files\n");
XX+ fprintf(stderr," - b : generate backtracking information to lex.backtrack\n");
XX+ fprintf(stderr," - c : compressed table, no equiv., no meta equiv.classes\n");
XX+ fprintf(stderr," e : equivalence classes\n");
XX+ fprintf(stderr," F : fast table\n");
XX+ fprintf(stderr," |f : full table\n");
XX+ fprintf(stderr," |m : meta equivalence classes\n");
XX+ fprintf(stderr," - d : generate debugging scanner\n");
XX+ fprintf(stderr," - F : fast table\n");
XX+ fprintf(stderr," - f : full (not compressed) table\n");
XX+ fprintf(stderr," - I : generate interactive scanner\n");
XX+ fprintf(stderr," - i : generate case-insensitive scanner\n");
XX+ fprintf(stderr," - L : supress #line directives\n");
XX+ /* fprintf(stderr," - n hexnum : generate scanner using <hexnum> as newline char.\n");*/
XX+ fprintf(stderr," - p : generate performance report to stderr\n");
XX+ fprintf(stderr," - S skeleton_path : file path for skeleton file\n");
XX+ fprintf(stderr," - s : suppress echo of unmatched scanner input to stdout\n");
XX+ fprintf(stderr," - T : run flex in trace mode\n");
XX+ #ifdef ATARI
XX+ fprintf(stderr," - t : place result on stdout instead of lex_yy.c\n");
XX+ #else
XX+ fprintf(stderr," - t : place result on stdout instead of lex.yy.c\n");
XX+ #endif
XX+ fprintf(stderr," - v : print statistics of generated scanner\n");
XX+ fprintf(stderr," default = -cem\n");
XX+ exit(status);
XX+ }
XX+
XX /* readin - read in the rules section of the input file(s)
XX *
XX * synopsis
X/
Xecho x - scan.l_cdiff
Xsed '/^X/s///' > scan.l_cdiff << '/'
XX*** Src_2.1/scan.l Thu Jun 30 19:42:00 1989
XX--- scan.l Thu Jul 2 22:19:26 1989
XX***************
XX*** 125,132 ****
XX
XX {SCNAME} RETURNNAME;
XX ^{OPTWS}\n ++linenum; /* allows blank lines in section 1 */
XX! \n ++linenum; return ( '\n' );
XX! . synerr( "illegal character" ); BEGIN(RECOVER);
XX
XX
XX <C_COMMENT>"*/" ECHO; BEGIN(0);
XX--- 125,136 ----
XX
XX {SCNAME} RETURNNAME;
XX ^{OPTWS}\n ++linenum; /* allows blank lines in section 1 */
XX! {OPTWS}\n ++linenum; return ( '\n' );
XX! . {
XX! synerr( "illegal character" );
XX! fprintf(stderr,"Char : $%x\n",yytext[yyleng-1]);
XX! BEGIN(RECOVER);
XX! }
XX
XX
XX <C_COMMENT>"*/" ECHO; BEGIN(0);
XX***************
XX*** 206,212 ****
XX <SECT2>^{OPTWS}\n ++linenum; /* allow blank lines in section 2 */
XX
XX /* this horrible mess of a rule matches indented lines which
XX! * do not contain "/*". We need to make the distinction because
XX * otherwise this rule will be taken instead of the rule which
XX * matches the beginning of comments like this one
XX */
XX--- 210,216 ----
XX <SECT2>^{OPTWS}\n ++linenum; /* allow blank lines in section 2 */
XX
XX /* this horrible mess of a rule matches indented lines which
XX! * do not contain '/','*'. We need to make the distinction because
XX * otherwise this rule will be taken instead of the rule which
XX * matches the beginning of comments like this one
XX */
X/
Xecho x - Makefile.tos
Xsed '/^X/s///' > Makefile.tos << '/'
XX# make file for "flex" tool
XX
XX# @(#) $Header: Makefile,v 2.3 89/06/20 17:27:12 vern Exp $ (LBL)
XX
XX# Porting considerations:
XX#
XX# For System V Unix machines, add -DSYS_V to CFLAGS.
XX# For Vax/VMS, add -DSYS_V to CFLAGS.
XX# For MS-DOS, add "-DMS_DOS -DSYS_V" to CFLAGS. Create \tmp if not present.
XX# You will also want to rename flex.skel to something with a three
XX# character extension, change SKELETON_FILE below appropriately,
XX# For Amiga, add "-DAMIGA -DSYS_V" to CFLAGS.
XX#
XX# A long time ago, flex was successfully built using Microsoft C and
XX# the following options: /AL, /stack:10000, -LARGE, -Ml, -Mt128, -DSYS_V
XX
XX
XX# the first time around use "make first_flex"
XX
XX# The following definitions must be set according to your compiler -
XX# examples for a Lattice Compiler with GST assembler and TURBO C with
XX# assembler are provided below and must just be updated (don't forget to
XX# update the linker argument extension files (*.lnk,*.tlk), too) :
XX#
XX#CCPATH = path to compiler directory without trailing \
XX#CHPATH = path to header files without trailing \
XX#CC = filename of the compiler
XX#CFLAGS = compiler option flags
XX#CIEXT = extension of C sources that should be used for input filenames
XX#ASMPATH = path to assembler directory without trailing \
XX#ASM = filename of the assembler
XX#AFLAGS = assembler option flags
XX#AIEXT = extension of assembler sources that should be used for assembler
XX# input filenames
XX#AEXT = general assembler filename extension
XX#LNKPATH = path to linker directory without trailing \
XX#LINK = filename of the linker
XX#LFLAG0 = first option (full pathname of C startupcode)
XX#LFLAG1 = further options + option flag for argument extension filename
XX#LFLAG2 = further options + option flag for output-filename
XX#LNKEXT = extension of linker argument extension file
XX#OIEXT = extension of objects that should be used for linker input files
XX#OEXT = general object file extension
XX
XX# Lattice definitions
XXCCPATH = d:\latt
XXCHPATH = d:\latt\include
XXCC = lc.ttp
XXCFLAGS = -h -n -cw -cc -i$(CHPATH)\ -g$(CCPATH)\ -dLATTICE -dATARI
XXCIEXT =
XXASMPATH = d:\gst
XXASM = assem.ttp
XXAFLAGS = -nolist -errors errors.out
XXAIEXT =
XXAEXT = .asm
XXLNKPATH = d:\gst
XXLINK = ld.ttp
XXLFLAG0 =
XXLFLAG1 = -with
XXLFLAG2 = -nolist -sec -mem 200 -prog
XXLNKEXT = .lnk
XXOIEXT =
XXOEXT = .bin
XX
XX# Turbo definitions
XX#CCPATH = d:\turbo
XX#CHPATH = d:\turbo\include
XX#CC = tcc.prg
XX#CFLAGS = -GJMPRZ -H=$(CHPATH)\ -w- -DTURBO -DATARI
XX#CIEXT = .c
XX#ASMPATH = d:\turbo
XX#ASM = mas.prg
XX#AFLAGS =
XX#AIEXT = .s
XX#AEXT = .s
XX#LNKPATH = d:\turbo
XX#LINK = tlink.ttp
XX#LFLAG0 = $(LNKPATH)\lib\tcstart.o
XX#LFLAG1 = -I=
XX#LFLAG2 = -O=
XX#LNKEXT = .tlk
XX#OIEXT = .o
XX#OEXT = .o
XX
XX# Other definitions
XX# (not used for Atari because of short argument string - defined in flexdef.h
XX
XXSKELETON_DIR = /usr/lib
XXSKELETON_FILE = flex.skel
XXSKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
XX
XX
XXFLEX = d:\etc\flex.ttp
XXFLEX_FLAGS = -is
XXYACC = d:\etc\yacc.ttp
XXRENAME = d:\bin\rename
XXDEL = d:\bin\del
XX
XX# Internal definitions
XXLNK = $(LNKPATH)\$(LINK)
XX
XXFLEXOBJS = \
XX ccl$(OEXT) \
XX dfa$(OEXT) \
XX ecs$(OEXT) \
XX gen$(OEXT) \
XX main$(OEXT) \
XX misc$(OEXT) \
XX nfa$(OEXT) \
XX parse$(OEXT) \
XX scan$(OEXT) \
XX sym$(OEXT) \
XX tblcmp$(OEXT) \
XX yylex$(OEXT)
XX
XXFLEX_C_SOURCES = \
XX ccl.c \
XX dfa.c \
XX ecs.c \
XX gen.c \
XX main.c \
XX misc.c \
XX nfa.c \
XX parse.c \
XX scan.c \
XX sym.c \
XX tblcmp.c \
XX yylex.c
XX
XXflex : $(FLEXOBJS)
XX $(LNK) $(LFLAG0) $(LFLAG1)flex$(LNKEXT) $(LFLAG2)flex.ttp
XX
XXfirst_flex:
XX cp initscan.c scan.c
XX make $(MFLAGS) flex
XX
XXparse.h parse.c : parse.y
XX $(YACC) -d parse.y
XX $(DEL) parse.c
XX $(RENAME) y_tab.c parse.c
XX $(DEL) parse.h
XX $(RENAME) y_tab.h parse.h
XX
XXscan.c : scan.l
XX $(FLEX) $(FLEX_FLAGS) scan.l
XX $(RENAME) lex_yy.c scan.c
XX
XXscan$(OEXT) : scan.c parse.h flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) scan$(CIEXT)
XX
XXmain$(OEXT) : main.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) main$(CIEXT)
XX
XXccl$(OEXT) : ccl.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) ccl$(CIEXT)
XX
XXdfa$(OEXT) : dfa.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) dfa$(CIEXT)
XX
XXecs$(OEXT) : ecs.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) ecs$(CIEXT)
XX
XXgen$(OEXT) : gen.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) gen$(CIEXT)
XX
XXmisc$(OEXT) : misc.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) misc$(CIEXT)
XX
XXnfa$(OEXT) : nfa.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) nfa$(CIEXT)
XX
XXparse$(OEXT) : parse.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) parse$(CIEXT)
XX
XXsym$(OEXT) : sym.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) sym$(CIEXT)
XX
XXtblcmp$(OEXT) : tblcmp.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) tblcmp$(CIEXT)
XX
XXyylex$(OEXT) : yylex.c flexdef.h
XX $(CCPATH)\$(CC) $(CFLAGS) yylex$(CIEXT)
XX
XXflex.man : flex.1
XX nroff -man flex.1 >flex.man
XX
XXlint : $(FLEX_C_SOURCES)
XX lint $(FLEX_C_SOURCES) > flex.lint
XX
XXdistrib :
XX mv scan.c initscan.c
XX chmod 444 initscan.c
XX $(MAKE) $(MFLAGS) clean
XX
XXclean :
XX rm -f core errs flex *$(OEXT) parse.c *.lint parse.h flex.man tags
XX
XXtags :
XX ctags $(FLEX_C_SOURCES)
XX
XXvms : flex.man
XX $(MAKE) $(MFLAGS) distrib
XX
XXtest :
XX $(FLEX) $(FLEX_FLAGS) scan.l
XX $(RENAME) lex_yy.c scan.ctest
XX cmp scan.c scan.ctest
XX
X/
Xecho x - Readme2
Xsed '/^X/s///' > Readme2 << '/'
XXThe changes: (1.7.89 RAL)
XX
XX - Bug fix: The original flex didn't like trailing spaces in exclusive start
XX condition lists ! If you add an trailing space to line 68 in scan.l
XX
XX "%x SECT2 SECT2PROLOG SECT3 CODEBLOCK PICKUPDEF SC CARETISBOL NUM QUOTE "
XX
XX you get a misleading error message:
XX
XX "Syntax error at line 69: bad start condition list"
XX
XX This bug can either be fixed in parse.y or in scan.l . I have chosen the
XX last because there the fix is minimal: Just change the rule (line 128)
XX
XX "\n" to "{OPTWS}\n"
XX
XX - Enhancements:
XX - new option "-?" that provides some help information about the other
XX flags (main.c)
XX - new option "-aTMPPATH" that allows a redefinition of the standard
XX path for the temporary file (this might be rather large >200k if
XX F,f options are selected). (main.c, flexdef.h (l.376))
XX - hexdump of illegal characters -- this proved to be a useful debugging
XX tool especialy if invisible control characters occur which weren't
XX covered by the rules. (scan.l fprintf statement line 129,...)
XX
XX - Patches due to TOS
XX - General: TOS uses CR,LF as end of line char., Flex wants only a single
XX LF as EOL char. Therefore all I/O must be translated using f* calls.
XX This is done everywhere besides the YY_INPUT macro (flex.skel (scan.c),
XX line 31) that uses a low level 'read'. This should be definitly changed
XX to fread, so that all I/O calls occur on the same level.
XX - the short action_file_name has been "flexXXXXXX.tmp", but that's too
XX much for TOS,MSDOS ! I changed it to "flexXXXX.tmp" in main.c (patch
XX covered by the -a option additions)
XX - some compilers don't like external names that are ambiguous within
XX the first 8 characters. I defined macros that change all these long
XX external names to names that are unique within the first 8 characters.
XX Just define SHORT_EXTERNAL_NAMES to use this feature (flexdef.h)
XX - some statement changes that some compiler don't like:
XX typedef *xxx[] -> typedef **xxx (flexdef.h.,l.308)
XX "/*" -> '/','*' within a comment in (scan.l, l.209)
XX - changed short "lexyy.c" to "lex_yy.c" what's more similar to the unix
XX "lex.yy.c" (main.c).
XX - a few further really compiler dependent changes provided with
XX #ifdef ATARI && LATTICE res. TURBO braces.
XX
XX - Additional TOS only files
XX - Makefile.tos: Common makefile for all TOS compilers. If you add further
XX ones please email me the new makefile.
XX - flex.lnk: Lattice - GST linker argument extension file
XX - flex.tlk: Turbo linker argument extension file
XX
XX
XXAdditional remarks:
XX
XXI didn't add a new initscan.c (= flex -ist scan.l). The current one is good
XXenough for a first compilation. With this first version of flex you can
XXrebuild your own scan.c and the final flex version !
XX
XXMinix ST :
XX - I had to "chmem =70000 cv" (>50000) to get flex linked
XX - 'memset' (PC 1.3, EFTH40,...) is necessary
XX - chmem =90000 flex may be sufficient
XX
XXMinix PC :
XX It should be possible to port Flex to Minix PC. The current sizes of flex
XX are:
XX Minix ST (ACK) Lattice (TOS) Turbo (TOS)
XX
XX size 75300 83305 57957
XX compilation time 22' 15' 3'40"
XX flex -is scan.l 1'49" 43" 30"
XX
XX The Minix ST size includes the bad generated code using only a subset of
XX the 68000 commands, long addresses only and a huge relocation table.
XX Therefore the PC size will be <64 k ! More serious is the fact that I had
XX to chmem =90000 flex to get scan.l converted to scan.c . But I never saw
XX a more complex lex source than scan.l -- so it should be possible to
XX reduce some array sizes without limitation for all day usage.
XX
XX No one volunteered yet for a Minix PC port -- but if someone will try it
XX I would provide him with a new scan.c and some hints.
XX
XXTOS:
XX Don't forget to adapt the flexskel path within flexdef.h !
XX
XX
X/
Xecho x - flex.lnk
Xsed '/^X/s///' > flex.lnk << '/'
XX*
XX*
XX* linker control file for flex.ttp
XX*
XX*
XX*
XXINPUT d:\latt\lib\startup.bin
XX*
XXINPUT ccl.bin
XXINPUT dfa.bin
XXINPUT ecs.bin
XXINPUT gen.bin
XXINPUT misc.bin
XXINPUT nfa.bin
XXINPUT parse.bin
XXINPUT sym.bin
XXINPUT tblcmp.bin
XXINPUT main.bin
XXINPUT yylex.bin
XXINPUT scan.bin
XX*
XXLIBRARY d:\latt\lib\clib.bin
XX
X/
Xecho x - flex.tlk
Xsed '/^X/s///' > flex.tlk << '/'
XXccl.o
XXdfa.o
XXecs.o
XXgen.o
XXmisc.o
XXnfa.o
XXparse.o
XXsym.o
XXtblcmp.o
XXyylex.o
XXmain.o
XXscan.o
XXd:\turbo\lib\tcstdlib.lib ; standard lib
XXd:\turbo\lib\tcextlib.lib ; extended lib
XXd:\turbo\lib\tctoslib.lib ; TOS lib
XXd:\turbo\lib\tcgemlib.lib ; AES and VDI lib
XX-S=200000
X/
END_OF_FILE
if test 29474 -ne `wc -c <'MISC/Atari.patches'`; then
echo shar: \"'MISC/Atari.patches'\" unpacked with wrong size!
fi
# end of 'MISC/Atari.patches'
fi
if test -f 'flex.1' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'flex.1'\"
else
echo shar: Extracting \"'flex.1'\" \(20799 characters\)
sed "s/^X//" >'flex.1' <<'END_OF_FILE'
X.TH FLEX 1 "26 May 1990" "Version 2.3"
X.SH NAME
Xflex - fast lexical analyzer generator
X.SH SYNOPSIS
X.B flex
X.B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton]
X.I [filename ...]
X.SH DESCRIPTION
X.I flex
Xis a tool for generating
X.I scanners:
Xprograms which recognized lexical patterns in text.
X.I flex
Xreads
Xthe given input files, or its standard input if no file names are given,
Xfor a description of a scanner to generate. The description is in
Xthe form of pairs
Xof regular expressions and C code, called
X.I rules. flex
Xgenerates as output a C source file,
X.B lex.yy.c,
Xwhich defines a routine
X.B yylex().
XThis file is compiled and linked with the
X.B -lfl
Xlibrary to produce an executable. When the executable is run,
Xit analyzes its input for occurrences
Xof the regular expressions. Whenever it finds one, it executes
Xthe corresponding C code.
X.LP
XFor full documentation, see
X.B flexdoc(1).
XThis manual entry is intended for use as a quick reference.
X.SH OPTIONS
X.I flex
Xhas the following options:
X.TP
X.B -b
XGenerate backtracking information to
X.I lex.backtrack.
XThis is a list of scanner states which require backtracking
Xand the input characters on which they do so. By adding rules one
Xcan remove backtracking states. If all backtracking states
Xare eliminated and
X.B -f
Xor
X.B -F
Xis used, the generated scanner will run faster.
X.TP
X.B -c
Xis a do-nothing, deprecated option included for POSIX compliance.
X.IP
X.B NOTE:
Xin previous releases of
X.I flex
X.B -c
Xspecified table-compression options. This functionality is
Xnow given by the
X.B -C
Xflag. To ease the the impact of this change, when
X.I flex
Xencounters
X.B -c,
Xit currently issues a warning message and assumes that
X.B -C
Xwas desired instead. In the future this "promotion" of
X.B -c
Xto
X.B -C
Xwill go away in the name of full POSIX compliance (unless
Xthe POSIX meaning is removed first).
X.TP
X.B -d
Xmakes the generated scanner run in
X.I debug
Xmode. Whenever a pattern is recognized and the global
X.B yy_flex_debug
Xis non-zero (which is the default), the scanner will
Xwrite to
X.I stderr
Xa line of the form:
X.nf
X
X --accepting rule at line 53 ("the matched text")
X
X.fi
XThe line number refers to the location of the rule in the file
Xdefining the scanner (i.e., the file that was fed to flex). Messages
Xare also generated when the scanner backtracks, accepts the
Xdefault rule, reaches the end of its input buffer (or encounters
Xa NUL; the two look the same as far as the scanner's concerned),
Xor reaches an end-of-file.
X.TP
X.B -f
Xspecifies (take your pick)
X.I full table
Xor
X.I fast scanner.
XNo table compression is done. The result is large but fast.
XThis option is equivalent to
X.B -Cf
X(see below).
X.TP
X.B -i
Xinstructs
X.I flex
Xto generate a
X.I case-insensitive
Xscanner. The case of letters given in the
X.I flex
Xinput patterns will
Xbe ignored, and tokens in the input will be matched regardless of case. The
Xmatched text given in
X.I yytext
Xwill have the preserved case (i.e., it will not be folded).
X.TP
X.B -n
Xis another do-nothing, deprecated option included only for
XPOSIX compliance.
X.TP
X.B -p
Xgenerates a performance report to stderr. The report
Xconsists of comments regarding features of the
X.I flex
Xinput file which will cause a loss of performance in the resulting scanner.
X.TP
X.B -s
Xcauses the
X.I default rule
X(that unmatched scanner input is echoed to
X.I stdout)
Xto be suppressed. If the scanner encounters input that does not
Xmatch any of its rules, it aborts with an error.
X.TP
X.B -t
Xinstructs
X.I flex
Xto write the scanner it generates to standard output instead
Xof
X.B lex.yy.c.
X.TP
X.B -v
Xspecifies that
X.I flex
Xshould write to
X.I stderr
Xa summary of statistics regarding the scanner it generates.
X.TP
X.B -F
Xspecifies that the
X.ul
Xfast
Xscanner table representation should be used. This representation is
Xabout as fast as the full table representation
X.ul
X(-f),
Xand for some sets of patterns will be considerably smaller (and for
Xothers, larger). See
X.B flexdoc(1)
Xfor details.
X.IP
XThis option is equivalent to
X.B -CF
X(see below).
X.TP
X.B -I
Xinstructs
X.I flex
Xto generate an
X.I interactive
Xscanner, that is, a scanner which stops immediately rather than
Xlooking ahead if it knows
Xthat the currently scanned text cannot be part of a longer rule's match.
XAgain, see
X.B flexdoc(1)
Xfor details.
X.IP
XNote,
X.B -I
Xcannot be used in conjunction with
X.I full
Xor
X.I fast tables,
Xi.e., the
X.B -f, -F, -Cf,
Xor
X.B -CF
Xflags.
X.TP
X.B -L
Xinstructs
X.I flex
Xnot to generate
X.B #line
Xdirectives in
X.B lex.yy.c.
XThe default is to generate such directives so error
Xmessages in the actions will be correctly
Xlocated with respect to the original
X.I flex
Xinput file, and not to
Xthe fairly meaningless line numbers of
X.B lex.yy.c.
X.TP
X.B -T
Xmakes
X.I flex
Xrun in
X.I trace
Xmode. It will generate a lot of messages to
X.I stdout
Xconcerning
Xthe form of the input and the resultant non-deterministic and deterministic
Xfinite automata. This option is mostly for use in maintaining
X.I flex.
X.TP
X.B -8
Xinstructs
X.I flex
Xto generate an 8-bit scanner.
XOn some sites, this is the default. On others, the default
Xis 7-bit characters. To see which is the case, check the verbose
X.B (-v)
Xoutput for "equivalence classes created". If the denominator of
Xthe number shown is 128, then by default
X.I flex
Xis generating 7-bit characters. If it is 256, then the default is
X8-bit characters.
X.TP
X.B -C[efmF]
Xcontrols the degree of table compression.
X.IP
X.B -Ce
Xdirects
X.I flex
Xto construct
X.I equivalence classes,
Xi.e., sets of characters
Xwhich have identical lexical properties.
XEquivalence classes usually give
Xdramatic reductions in the final table/object file sizes (typically
Xa factor of 2-5) and are pretty cheap performance-wise (one array
Xlook-up per character scanned).
X.IP
X.B -Cf
Xspecifies that the
X.I full
Xscanner tables should be generated -
X.I flex
Xshould not compress the
Xtables by taking advantages of similar transition functions for
Xdifferent states.
X.IP
X.B -CF
Xspecifies that the alternate fast scanner representation (described in
X.B flexdoc(1))
Xshould be used.
X.IP
X.B -Cm
Xdirects
X.I flex
Xto construct
X.I meta-equivalence classes,
Xwhich are sets of equivalence classes (or characters, if equivalence
Xclasses are not being used) that are commonly used together. Meta-equivalence
Xclasses are often a big win when using compressed tables, but they
Xhave a moderate performance impact (one or two "if" tests and one
Xarray look-up per character scanned).
X.IP
XA lone
X.B -C
Xspecifies that the scanner tables should be compressed but neither
Xequivalence classes nor meta-equivalence classes should be used.
X.IP
XThe options
X.B -Cf
Xor
X.B -CF
Xand
X.B -Cm
Xdo not make sense together - there is no opportunity for meta-equivalence
Xclasses if the table is not being compressed. Otherwise the options
Xmay be freely mixed.
X.IP
XThe default setting is
X.B -Cem,
Xwhich specifies that
X.I flex
Xshould generate equivalence classes
Xand meta-equivalence classes. This setting provides the highest
Xdegree of table compression. You can trade off
Xfaster-executing scanners at the cost of larger tables with
Xthe following generally being true:
X.nf
X
X slowest & smallest
X -Cem
X -Cm
X -Ce
X -C
X -C{f,F}e
X -C{f,F}
X fastest & largest
X
X.fi
X.IP
X.B -C
Xoptions are not cumulative; whenever the flag is encountered, the
Xprevious -C settings are forgotten.
X.TP
X.B -Sskeleton_file
Xoverrides the default skeleton file from which
X.I flex
Xconstructs its scanners. You'll never need this option unless you are doing
X.I flex
Xmaintenance or development.
X.SH SUMMARY OF FLEX REGULAR EXPRESSIONS
XThe patterns in the input are written using an extended set of regular
Xexpressions. These are:
X.nf
X
X x match the character 'x'
X . any character except newline
X [xyz] a "character class"; in this case, the pattern
X matches either an 'x', a 'y', or a 'z'
X [abj-oZ] a "character class" with a range in it; matches
X an 'a', a 'b', any letter from 'j' through 'o',
X or a 'Z'
X [^A-Z] a "negated character class", i.e., any character
X but those in the class. In this case, any
X character EXCEPT an uppercase letter.
X [^A-Z\\n] any character EXCEPT an uppercase letter or
X a newline
X r* zero or more r's, where r is any regular expression
X r+ one or more r's
X r? zero or one r's (that is, "an optional r")
X r{2,5} anywhere from two to five r's
X r{2,} two or more r's
X r{4} exactly 4 r's
X {name} the expansion of the "name" definition
X (see above)
X "[xyz]\\"foo"
X the literal string: [xyz]"foo
X \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
X then the ANSI-C interpretation of \\x.
X Otherwise, a literal 'X' (used to escape
X operators such as '*')
X \\123 the character with octal value 123
X \\x2a the character with hexadecimal value 2a
X (r) match an r; parentheses are used to override
X precedence (see below)
X
X
X rs the regular expression r followed by the
X regular expression s; called "concatenation"
X
X
X r|s either an r or an s
X
X
X r/s an r but only if it is followed by an s. The
X s is not part of the matched text. This type
X of pattern is called as "trailing context".
X ^r an r, but only at the beginning of a line
X r$ an r, but only at the end of a line. Equivalent
X to "r/\\n".
X
X
X <s>r an r, but only in start condition s (see
X below for discussion of start conditions)
X <s1,s2,s3>r
X same, but in any of start conditions s1,
X s2, or s3
X
X
X <<EOF>> an end-of-file
X <s1,s2><<EOF>>
X an end-of-file when in start condition s1 or s2
X
X.fi
XThe regular expressions listed above are grouped according to
Xprecedence, from highest precedence at the top to lowest at the bottom.
XThose grouped together have equal precedence.
X.LP
XSome notes on patterns:
X.IP -
XNegated character classes
X.I match newlines
Xunless "\\n" (or an equivalent escape sequence) is one of the
Xcharacters explicitly present in the negated character class
X(e.g., "[^A-Z\\n]").
X.IP -
XA rule can have at most one instance of trailing context (the '/' operator
Xor the '$' operator). The start condition, '^', and "<<EOF>>" patterns
Xcan only occur at the beginning of a pattern, and, as well as with '/' and '$',
Xcannot be grouped inside parentheses. The following are all illegal:
X.nf
X
X foo/bar$
X foo|(bar$)
X foo|^bar
X <sc1>foo<sc2>bar
X
X.fi
X.SH SUMMARY OF SPECIAL ACTIONS
XIn addition to arbitrary C code, the following can appear in actions:
X.IP -
X.B ECHO
Xcopies yytext to the scanner's output.
X.IP -
X.B BEGIN
Xfollowed by the name of a start condition places the scanner in the
Xcorresponding start condition.
X.IP -
X.B REJECT
Xdirects the scanner to proceed on to the "second best" rule which matched the
Xinput (or a prefix of the input).
X.B yytext
Xand
X.B yyleng
Xare set up appropriately. Note that
X.B REJECT
Xis a particularly expensive feature in terms scanner performance;
Xif it is used in
X.I any
Xof the scanner's actions it will slow down
X.I all
Xof the scanner's matching. Furthermore,
X.B REJECT
Xcannot be used with the
X.I -f
Xor
X.I -F
Xoptions.
X.IP
XNote also that unlike the other special actions,
X.B REJECT
Xis a
X.I branch;
Xcode immediately following it in the action will
X.I not
Xbe executed.
X.IP -
X.B yymore()
Xtells the scanner that the next time it matches a rule, the corresponding
Xtoken should be
X.I appended
Xonto the current value of
X.B yytext
Xrather than replacing it.
X.IP -
X.B yyless(n)
Xreturns all but the first
X.I n
Xcharacters of the current token back to the input stream, where they
Xwill be rescanned when the scanner looks for the next match.
X.B yytext
Xand
X.B yyleng
Xare adjusted appropriately (e.g.,
X.B yyleng
Xwill now be equal to
X.I n
X).
X.IP -
X.B unput(c)
Xputs the character
X.I c
Xback onto the input stream. It will be the next character scanned.
X.IP -
X.B input()
Xreads the next character from the input stream (this routine is called
X.B yyinput()
Xif the scanner is compiled using
X.B C++).
X.IP -
X.B yyterminate()
Xcan be used in lieu of a return statement in an action. It terminates
Xthe scanner and returns a 0 to the scanner's caller, indicating "all done".
X.IP
XBy default,
X.B yyterminate()
Xis also called when an end-of-file is encountered. It is a macro and
Xmay be redefined.
X.IP -
X.B YY_NEW_FILE
Xis an action available only in <<EOF>> rules. It means "Okay, I've
Xset up a new input file, continue scanning".
X.IP -
X.B yy_create_buffer( file, size )
Xtakes a
X.I FILE
Xpointer and an integer
X.I size.
XIt returns a YY_BUFFER_STATE
Xhandle to a new input buffer large enough to accomodate
X.I size
Xcharacters and associated with the given file. When in doubt, use
X.B YY_BUF_SIZE
Xfor the size.
X.IP -
X.B yy_switch_to_buffer( new_buffer )
Xswitches the scanner's processing to scan for tokens from
Xthe given buffer, which must be a YY_BUFFER_STATE.
X.IP -
X.B yy_delete_buffer( buffer )
Xdeletes the given buffer.
X.SH VALUES AVAILABLE TO THE USER
X.IP -
X.B char *yytext
Xholds the text of the current token. It may not be modified.
X.IP -
X.B int yyleng
Xholds the length of the current token. It may not be modified.
X.IP -
X.B FILE *yyin
Xis the file which by default
X.I flex
Xreads from. It may be redefined but doing so only makes sense before
Xscanning begins. Changing it in the middle of scanning will have
Xunexpected results since
X.I flex
Xbuffers its input. Once scanning terminates because an end-of-file
Xhas been seen,
X.B
Xvoid yyrestart( FILE *new_file )
Xmay be called to point
X.I yyin
Xat the new input file.
X.IP -
X.B FILE *yyout
Xis the file to which
X.B ECHO
Xactions are done. It can be reassigned by the user.
X.IP -
X.B YY_CURRENT_BUFFER
Xreturns a
X.B YY_BUFFER_STATE
Xhandle to the current buffer.
X.SH MACROS THE USER CAN REDEFINE
X.IP -
X.B YY_DECL
Xcontrols how the scanning routine is declared.
XBy default, it is "int yylex()", or, if prototypes are being
Xused, "int yylex(void)". This definition may be changed by redefining
Xthe "YY_DECL" macro. Note that
Xif you give arguments to the scanning routine using a
XK&R-style/non-prototyped function declaration, you must terminate
Xthe definition with a semi-colon (;).
X.IP -
XThe nature of how the scanner
Xgets its input can be controlled by redefining the
X.B YY_INPUT
Xmacro.
XYY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
Xaction is to place up to
X.I max_size
Xcharacters in the character array
X.I buf
Xand return in the integer variable
X.I result
Xeither the
Xnumber of characters read or the constant YY_NULL (0 on Unix systems)
Xto indicate EOF. The default YY_INPUT reads from the
Xglobal file-pointer "yyin".
XA sample redefinition of YY_INPUT (in the definitions
Xsection of the input file):
X.nf
X
X %{
X #undef YY_INPUT
X #define YY_INPUT(buf,result,max_size) \\
X { \\
X int c = getchar(); \\
X result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
X }
X %}
X
X.fi
X.IP -
XWhen the scanner receives an end-of-file indication from YY_INPUT,
Xit then checks the
X.B yywrap()
Xfunction. If
X.B yywrap()
Xreturns false (zero), then it is assumed that the
Xfunction has gone ahead and set up
X.I yyin
Xto point to another input file, and scanning continues. If it returns
Xtrue (non-zero), then the scanner terminates, returning 0 to its
Xcaller.
X.IP
XThe default
X.B yywrap()
Xalways returns 1. Presently, to redefine it you must first
X"#undef yywrap", as it is currently implemented as a macro. It is
Xlikely that
X.B yywrap()
Xwill soon be defined to be a function rather than a macro.
X.IP -
XYY_USER_ACTION
Xcan be redefined to provide an action
Xwhich is always executed prior to the matched rule's action.
X.IP -
XThe macro
X.B YY_USER_INIT
Xmay be redefined to provide an action which is always executed before
Xthe first scan.
X.IP -
XIn the generated scanner, the actions are all gathered in one large
Xswitch statement and separated using
X.B YY_BREAK,
Xwhich may be redefined. By default, it is simply a "break", to separate
Xeach rule's action from the following rule's.
X.SH FILES
X.TP
X.I flex.skel
Xskeleton scanner.
X.TP
X.I lex.yy.c
Xgenerated scanner (called
X.I lexyy.c
Xon some systems).
X.TP
X.I lex.backtrack
Xbacktracking information for
X.B -b
Xflag (called
X.I lex.bck
Xon some systems).
X.TP
X.B -lfl
Xlibrary with which to link the scanners.
X.SH "SEE ALSO"
X.LP
Xflexdoc(1), lex(1), yacc(1), sed(1), awk(1).
X.LP
XM. E. Lesk and E. Schmidt,
X.I LEX - Lexical Analyzer Generator
X.SH DIAGNOSTICS
X.I reject_used_but_not_detected undefined
Xor
X.LP
X.I yymore_used_but_not_detected undefined -
XThese errors can occur at compile time. They indicate that the
Xscanner uses
X.B REJECT
Xor
X.B yymore()
Xbut that
X.I flex
Xfailed to notice the fact, meaning that
X.I flex
Xscanned the first two sections looking for occurrences of these actions
Xand failed to find any, but somehow you snuck some in (via a #include
Xfile, for example). Make an explicit reference to the action in your
X.I flex
Xinput file. (Note that previously
X.I flex
Xsupported a
X.B %used/%unused
Xmechanism for dealing with this problem; this feature is still supported
Xbut now deprecated, and will go away soon unless the author hears from
Xpeople who can argue compellingly that they need it.)
X.LP
X.I flex scanner jammed -
Xa scanner compiled with
X.B -s
Xhas encountered an input string which wasn't matched by
Xany of its rules.
X.LP
X.I flex input buffer overflowed -
Xa scanner rule matched a string long enough to overflow the
Xscanner's internal input buffer (16K bytes - controlled by
X.B YY_BUF_MAX
Xin "flex.skel").
X.LP
X.I scanner requires -8 flag -
XYour scanner specification includes recognizing 8-bit characters and
Xyou did not specify the -8 flag (and your site has not installed flex
Xwith -8 as the default).
X.LP
X.I
Xfatal flex scanner internal error--end of buffer missed -
XThis can occur in an scanner which is reentered after a long-jump
Xhas jumped out (or over) the scanner's activation frame. Before
Xreentering the scanner, use:
X.nf
X
X yyrestart( yyin );
X
X.fi
X.LP
X.I too many %t classes! -
XYou managed to put every single character into its own %t class.
X.I flex
Xrequires that at least one of the classes share characters.
X.SH AUTHOR
XVern Paxson, with the help of many ideas and much inspiration from
XVan Jacobson. Original version by Jef Poskanzer.
X.LP
XSee flexdoc(1) for additional credits and the address to send comments to.
X.SH DEFICIENCIES / BUGS
X.LP
XSome trailing context
Xpatterns cannot be properly matched and generate
Xwarning messages ("Dangerous trailing context"). These are
Xpatterns where the ending of the
Xfirst part of the rule matches the beginning of the second
Xpart, such as "zx*/xy*", where the 'x*' matches the 'x' at
Xthe beginning of the trailing context. (Note that the POSIX draft
Xstates that the text matched by such patterns is undefined.)
X.LP
XFor some trailing context rules, parts which are actually fixed-length are
Xnot recognized as such, leading to the abovementioned performance loss.
XIn particular, parts using '|' or {n} (such as "foo{3}") are always
Xconsidered variable-length.
X.LP
XCombining trailing context with the special '|' action can result in
X.I fixed
Xtrailing context being turned into the more expensive
X.I variable
Xtrailing context. For example, this happens in the following example:
X.nf
X
X %%
X abc |
X xyz/def
X
X.fi
X.LP
XUse of unput() invalidates yytext and yyleng.
X.LP
XUse of unput() to push back more text than was matched can
Xresult in the pushed-back text matching a beginning-of-line ('^')
Xrule even though it didn't come at the beginning of the line
X(though this is rare!).
X.LP
XPattern-matching of NUL's is substantially slower than matching other
Xcharacters.
X.LP
X.I flex
Xdoes not generate correct #line directives for code internal
Xto the scanner; thus, bugs in
X.I flex.skel
Xyield bogus line numbers.
X.LP
XDue to both buffering of input and read-ahead, you cannot intermix
Xcalls to <stdio.h> routines, such as, for example,
X.B getchar(),
Xwith
X.I flex
Xrules and expect it to work. Call
X.B input()
Xinstead.
X.LP
XThe total table entries listed by the
X.B -v
Xflag excludes the number of table entries needed to determine
Xwhat rule has been matched. The number of entries is equal
Xto the number of DFA states if the scanner does not use
X.B REJECT,
Xand somewhat greater than the number of states if it does.
X.LP
X.B REJECT
Xcannot be used with the
X.I -f
Xor
X.I -F
Xoptions.
X.LP
XSome of the macros, such as
X.B yywrap(),
Xmay in the future become functions which live in the
X.B -lfl
Xlibrary. This will doubtless break a lot of code, but may be
Xrequired for POSIX-compliance.
X.LP
XThe
X.I flex
Xinternal algorithms need documentation.
END_OF_FILE
if test 20799 -ne `wc -c <'flex.1'`; then
echo shar: \"'flex.1'\" unpacked with wrong size!
fi
# end of 'flex.1'
fi
echo shar: End of archive 4 \(of 10\).
cp /dev/null ark4isdone
MISSING=""
for I in 1 2 3 4 5 6 7 8 9 10 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have unpacked all 10 archives.
rm -f ark[1-9]isdone ark[1-9][0-9]isdone
else
echo You still must unpack the following archives:
echo " " ${MISSING}
fi
exit 0
exit 0 # Just in case...
--
Please send comp.sources.unix-related mail to rsalz at uunet.uu.net.
Use a domain-based address or give alternate paths, or you may lose out.
More information about the Comp.sources.unix
mailing list