Perhaps the most common format of data for machine learning is text files. Often data is too large to fit in memory; this is sometimes referred to as big data. But do you need to load the whole data into memory? Maybe you could at least pre-process it line by line. We show how to do this with Python. Prepare to read and possibly write some code.
The most common format for text files is probably CSV. For sparse data, libsvm format is popular. Both can be processed using csv module in Python.
import csv i_f = open( input_file, 'r' ) reader = csv.reader( i_f )
For libsvm you just set the delimiter to space:
reader = csv.reader( i_f, delimiter = ' ' )
Then you go over the file contents. Each line is a list of strings:
for line in reader: # do something with the line, for example: label = float( line ) # .... writer.writerow( line )
If you need to do a second pass, you just rewind the input file:
i_f.seek( 0 ) for line in reader: # more stuff
We’ll demonstrate the idea using the split.py script as an example. You might have seen it already. It’s purpose is to randomly split the file into two, such that some lines from the original file go to the first ouput file and the rest go to the second. This is useful for creating training and testing set for validation.
We’d like to specify input and output files on the command line, like this:
python split.py train.csv train_v.csv test_v.csv
We’d also like to specify the probability of writing to the first file, so that for example 90% go to the train set and the rest to the test set:
python split.py train.csv train_v.csv test_v.csv 0.9
Let’s get to it. We need to import a few modules and read the file names:
import csv import sys import random input_file = sys.argv output_file1 = sys.argv output_file2 = sys.argv
If a user doesn’t specify P, it will be 0.9 by default:
try: P = float( sys.argv ) except IndexError: P = 0.9
It might be useful to be able to split the file again in the future in exactly the same way. Let’s say that you have some data and split it. Then you convert the data to some other format and again need to split it the same way so that you can compare the results achieved with different formats but the same split. Otherwise we might be comparing apples to oranges and that’s something we probably don’t want.
To get the same split every time, we’ll seed a random number generator. You give it a seed - an arbitrary string - and it will behave exactly the same every time. That’s because it’s not really random. So we’d like to be able to specify a seed on the command line as the final argument:
python split.py train.csv train_v.csv test_v.csv 0.9 a_random_seed
try: seed = sys.argv except IndexError: seed = None if seed: random.seed( seed )
Readers and writers
Let’s open the files and create a CSV reader and two writers. If you’re on Windows, it’s important to open the files for writing in binary mode (
'wb'), otherwise you might get some extra new lines.
i = open( input_file ) o1 = open( output_file1, 'wb' ) o2 = open( output_file2, 'wb' ) reader = csv.reader( i ) writer1 = csv.writer( o1 ) writer2 = csv.writer( o2 )
By the way, in this example we aren’t touching the contents of a line, so we don’t really need the csv module. Something like this would be enough:
i = open( input_file ) o = open( output_file, 'wb' ) for line in i: o.write( line )
Some files have headers in the first line. If that’s the case with your data, you can read the first line and write it to both output files:
headers = reader.next() writer1.writerow( headers ) writer2.writerow( headers )
Or maybe discard it.
After all this preparation we are ready for the final step: the loop. We read a line and then (pseudo)randomly decide whether to write it to one file or the other. The
random() function in the
random module returns a floating point number in the range [0.0, 1.0). We compare this number to P and on this basis decide which file to write to.
for line in reader: r = random.random() if r > P: writer2.writerow( line ) else: writer1.writerow( line )
Note that it’s an inexact method of splitting: if you have a thousand lines, you’ll get roughly 900 in the first file and roughly 100 in the second with P = 0.9.
An exact split
If you wanted to split the file exactly 900/100, here’s a way to do it. You shuffle integers from 0 to 999 and take the first 100.
n = 1000 indexes = random.sample( xrange( n ), n ) indexes = indexes[:100]
xrange() is just like
range(), only marginally more efficient. Basically it will give you integers from 0 to n-1, which we sample without replacement. In other words, we shuffle the list and divide it into two parts.
Then you do a second pass over the file. If a line’s number is in selected 100, you write to the second file; if not, to the first.
We use the described template extensively. It is usually quite fast, needs a little memory and can deal with files larger than available memory, unlike R or Matlab. If you want more examples, look through our Github repos and be sure to check out phraug.