Text Filtering in Automator

It sounds like something for grep and regular expressions. While I might be able to figure it out, eventually, you might find better response on the [Unix forum|http://discussions.apple.com/forum.jspa?forumID=735] or the [Applescript forum|http://discussions.apple.com/forum.jspa?forumID=724].

TextWrangler ought to work--if I'm correctly understanding the document format.

Open the source document in TextWrangler, then

Text > Process Lines Containing . . .
--Find lines containing C|T|A|G
--check "Use grep" and "Copy to new document"
--click Process

The new document contains only those lines containing any of the characters C, T, A, or G.

If you need to restrict the selection further, solely to lines of precisely 100 characters--that is, not lines of 0-99 or 101+ characters--that's going to take more filtering and a clearer description of the document format.

http://www.barebones.com/products/textwrangler/

Yes, the question did look like a bio problem! It might be easier to delete everything that's not a DNA sequence.

In TextWrangler, it would be three find/replace commands:

1. To delete lines that end in #
2. To delete lines that end in /2
3. To delete the remaining + signs that are on lines of their own

In TW, it's Search > Find, then in the dialogue box (with both Grep and Wrap around checked):

1. Find: .*#$
Replace:
2. Find: .*/2$
Replace:
3. Find: \+
Replace:

In each case, the replace field is left blank.

A test on the above data results in:



TGATCATTTGATTAATACTGACATGTAGACAAGAAGAAAAGTATGTTTCATGCTATTTTGAGTAACTTCCATTTAGAAGC CTACTCCTGAGCACAACATT



AATGTTATTAAAAATGGACACCTTTTTCTCACACATTCAGTTTCATTGTCTCGCACCCCATCGTTTTACTTTTCTTCCTT CAGAAAATGATAAATGTGGG



CAGATCAGAATAATTTTTGTGTTATGTACGTGTAAGAAAACATAGCTATTATGATATGGAAACTAGGAGTGAAATATGAG GAATTTGTGACTTTTCTGAA

If that's too much space between the remaining strings, you can use TW's find/replace to change three returns to one with

Find: \r\r\r\
Replace: \r

Are your examples really as you posted them, or are they each one paragraph (separated by a return)? If the metadata and base pair string is all on one line, you could also just use a script action to strip everything but the last 100 characters of a line.

A bit of Googling suggests we're looking at a fragment of a version (?) of this 77M dataset:

http://galaxy.rgenetics.org/datasets/1562201ac260fe38/display?to_ext=fastqsanger

It looks a fairly simple matter to massage it into shape to retain the relevant sequences in every fourth line starting with 2; delete the rest; and put a blank line between base-pair sequences.

1. Find: EAS
Replace:
{Delete a specific nonrelevant, document-wide string with an A in it}

2. Find: [^ACTG\+] {That should be left squarebracket caret ACTG\+ right squarebracket, but the board is garbling it}
Replace:
{Delete every remaining character except A C T G +}

3. Find: \+
Replace: \r\r
{Replace the + delimiter with two returns}



Step 2 might take awhile on over a million lines. You can speed it up by checking "Case sensitive" (along with "Grep") in the find/replace dialogue box.

Grep is your friend.

Paragraphs are separated by returns, so if each one of those was a paragraph you could just get the last 100 characters. There are several lines in each entry though, so another approach is needed. I would avoid using Automator for files this large, since it does a lot of caching and swapping around behind the scenes that really slow things down.

Give the following AppleScript a try. It uses a shell script to get lines that contain "C", "T", "G", and "\+" characters (the metadata lines can contain an "A", so I just skipped it), and then replaces the "\+" with a return. The resulting text is written to a file named "results.txt" on your desktop. I tested it with the file referenced by mns579, and it seems to work OK (the file was a bit large to count the results).

<pre style="
font-family: Monaco, 'Courier New', Courier, monospace;
font-size: 10px;
font-weight: normal;
margin: 0px;
padding: 5px;
border: 1px solid #000000;
width: 720px;
color: #000000;
background-color: #DAFFB6;
overflow: auto;"
title="this text can be pasted into the AppleScript Editor">
set input to POSIX path of (choose file)
set output to POSIX path of (((path to desktop folder) as text) & "results.txt")

do shell script "grep '[CTG+]' " & quoted form of input & " | sed 's/+/" & return & "/g' > " & quoted form of output
</pre>

As usual, red_menace offers an elegant solution. And it looks right to me: 1,235,385 lines go in, with the final line a blank; 308,846 ((1,235,385-1)/4) come out. Numerically at least, the script picks up one-fourth of the lines, which is what the pattern suggests should be the desired result.

Another way that also works with the posted text is to exclude all of the lines that have various metadata/mask characters, for example:

do shell script "grep -v '[!@#$%&?/.:]' " & quoted form of input & " | sed 's/+/" & return & "/g' > " & quoted form of output

My greppage isn't all that great, so I am sure there are other methods.

Using the POSIX path of your file name alters it by adding a leading slash, so your path isn't being set correctly (the original use is to convert from a Finder path). Scripts run from the Terminal can't have user interaction (e.g. the choose file dialog), so to use it there you can cheat a bit by asking another application to do that part:

tell application "System Events" to set input to POSIX path of (choose file)

You can also use the commands directly, dragging the various files to the Terminal window (this pastes in the path to the file), or use a here-document.

I didn't think you were familiar with the Terminal though, which is why I used an AppleScript. You can just paste the following script into the *AppleScript Editor* application and run it from there, or save it as an application. The script can also be used in an Automator workflow (for example as a Service), although as I mentioned earlier Automator chokes pretty bad on large files if you were wanting to use the results in another action. Post back with more details about your workflow to to see about wrapping it around the shell script.

<pre style="
font-family: Monaco, 'Courier New', Courier, monospace;
font-size: 10px;
font-weight: normal;
margin: 0px;
padding: 5px;
border: 1px solid #000000;
width: 720px;
color: #000000;
background-color: #DAFFB6;
overflow: auto;"
title="this text can be pasted into the AppleScript Editor">
set input to POSIX path of (choose file)
set output to POSIX path of (((path to desktop folder) as text) & "results.txt")

do shell script "grep -v '[!@#$%&?/.:]' " & quoted form of input & " | sed 's/+/" & return & "/g' > " & quoted form of output

-- do shell script "grep -E '(([ACTG]{6,})|(^\+$))' " & quoted form of input & " | sed 's/+/" & return & "/g' > " & quoted form of output
</pre>

In the above script I included a couple of different do shell scripts in case something slips through (just use one of them at a time, though). The first one uses an option that reverses the grep matching (it selects lines that do not match) and looks for lines that contain various characters that are only in the metadata lines. It then pipes the results to sed, which converts the lines that are "+" (this character was not included in the previous search) into an extra return.

The second do shell script (commented) is another one that I was playing with. it differs in that grep looks for lines that contain 6 or more of any of the base pair characters, and lines that consist of only a single "+" character.

I've tested both of these on the example text you posted earlier, and the large file that mns579 found - both methods appear to work OK (I didn't go through the entire results for the large document).