I am trying to run a Parsec
parser over a whole bunch of small files, and getting an error saying I have too many open files. I understand that I need to use strict IO, but I'm not sure how to do that. This is the problematic code:
files = getDirectoryContents historyFolder
hands :: IO [Either ParseError [Hand]]
hands = join $ sequence <$> parseFromFile (many hand) <<$>> files
Note: my <<$>>
function is this:
(<<$>>) :: (Functor f1, Functor f2) => (a -> b) -> f1 (f2 a) -> f1 (f2 b)
a <<$>> b = (a <$>) <$> b
I don't know what your
parseFromFile
function looks like right now (probably a good idea to include that in the question), but I'm guessing you're usingPrelude.readFile
, which as @Markus1189 points out includes lazy I/O. To get to strict I/O, you just need a strictreadFile
, such as Data.Text.IO.readFile.A streaming data library like
pipes
orconduit
would allow you to avoid reading the entire file into memory at once, though- to my knowledge- parsec doesn't provide a streaming interface to allow this to happen. attoparsec, on the other hand, does include such a streaming interface, and both pipes and conduit have attoparsec adapter libraries (e.g., Data.Conduit.Attoparsec).tl;dr: You probably just need the following helper function: