# How to append something to the end of everyline of a very large file



## TeresaModesta (Oct 29, 2013)

Hi,

I have a text file that is upwards of 50GB and I want to change it so every line has the following literal appended; -


,12345.67

Is this easily written in a batch file and am I likely to run into any memory issues whilst running given the size of my file?


If my requirement was altered to append a randomly generated decimal of 2 digits between 0.01 and 1000000.00 with a comma would the result run massively slower?


thanks,

Teresa.


----------



## Squashman (Apr 4, 2003)

You will run out of memory.
I use a 3rd party utility to append a static variable to the end of every line of a file. I dont think it would be possible to do a random number with it. 
Utility is called Swiss File Knife. 
It truly is just like it sounds. Has a lot of neat utilities built into it.


----------



## valis (Sep 24, 2004)

Squashman, I subscribed to this thread specifically to see your answer, as I suspected you'd be by......and thanks for that, as I'd not heard of that one. :up:


----------



## foxidrive (Oct 20, 2012)

TeresaModesta said:


> Hi,
> 
> I have a text file that is upwards of 50GB and I want to change it so every line has the following literal appended; -
> 
> ...


As the file would be opened and read line by line, then writing to another file, memory wouldn't be an issue. The second file would also take up the 50 GB on your hard drive while it is being created, and it would probably be rather slow. Doing this on an SSD would be a good option to improve speed. A VBS script may work better.



> If my requirement was altered to append a randomly generated decimal of 2 digits between 0.01 and 1000000.00 with a comma would the result run massively slower?


I'm not sure what you are asking here - a 2 digit number only has two digits.
Shows some examples of the number you want.


----------



## Squashman (Apr 4, 2003)

```
C:\Users\Squashman\Desktop>dir dummy1.txt
 Volume in drive C has no label.
 Volume Serial Number is 24B6-8062

 Directory of C:\Users\Squashman\Desktop

10/30/2013  09:58 PM     1,075,838,976 dummy1.txt
               1 File(s)  1,075,838,976 bytes
               0 Dir(s)  195,447,447,552 bytes free

C:\Users\Squashman\Desktop>for /f %G in (dummy1.txt) do echo %G
Not enough storage is available to process this command.
```


----------



## foxidrive (Oct 20, 2012)

Confirmed here in Windows 8 32 bit.

A 926 MB file works but it seems to be a 1 GB limitation.

A VBS script works with a 1.8 GB file and it wrote a 2.86 GB file - the OP can try a 50 GB file with this. 
I'm using a 32 bit OS with only 3.5 GB RAM available so considering there was no out-of-memory error then it may work with a larger file too.

Read-write-a-file.vbs


```
Const ForReading = 1, ForWriting = 2

  infile = "infile.txt"   '<= Set input file here
  outfile = "outfile.txt" '<= Set output file here

  Set fso = CreateObject("Scripting.FileSystemObject")
  Set f1 = fso.OpenTextFile(infile, ForReading)
  Set f2 = fso.OpenTextFile(outfile, ForWriting, True)

  Do While not F1.AtEndOfStream
  line=F1.readline
  f2.writeline line+ " end of line text"
  Loop
  f1.Close
  f2.Close
```


----------



## Squashman (Apr 4, 2003)

It is really dependent on how much free memory you have. I used to run my append record routine in pure batch years ago and I knew on my old system which was Window XP with 2GB of ram I could at most read in a 600MB file.

SFK was a life saver.

But then I started learning Powershell and it is more like VBscript where it is truly reading the file line by line but Batch loads the entire file into memory before it starts processing it. In Powershell you can even specify how many lines of input you want it to read into memory before it starts processing. Which is actually much faster than reading line by line.


----------



## Squashman (Apr 4, 2003)

Here is the code I use with Swiss File Knife

```
sfk addtail <"InputFile.txt" >>OutputFile.txt -noblank "AppendingThisToTheEnd"
```


----------



## foxidrive (Oct 20, 2012)

Squashman said:


> In Powershell you can even specify how many lines of input you want it to read into memory before it starts processing. Which is actually much faster than reading line by line.


That would make a big difference, setting a large buffer.

I've had swiss file knife for a very long time, but haven't actually used it often. It's certainly has a long list of capabilities.

For the random number generation: VBS should work and if it's a one off transformation then the time taken isn't really an issue.

I'd like to see your Powershell solution for this task though - I'm very much inexperienced with Powershell.


----------



## Squashman (Apr 4, 2003)

Wrote it about 5 years ago but don't know what happened to it. Corporate wasn't letting me install Powershell on all the computers so I just said screw it and kept using SFK. That was 3 computers ago at work. I am sure I have it backed up some where. Just not sure where. 

Powershell is amazing cool once you learn it. I have sort of forgotten everything in the past few years.


----------

