So I've been working on some scraping scripts with Ruby, using the gem watir lately. I used to put the results in a .json file but I recently started working with MySQL2. So here's the deal:
I have created a db called scrap, with a table called links, which has 3 columns: id, job, link. The script I'm working on for this is very simple: it makes a query on google search, takes the URLs of all the links, then goes to the next page. So here's the boy:
require "watir"
require "json"
require "mysql2"
b = Watir::Browser.new(:firefox)
client = Mysql2::Client.new(:host => "localhost", :username => "root", :password => "xxx", :database => "scrap", :encoding => "utf8")
def Launch(b)
job = "cabinet avocat"
ville = "paris"
k = 0
j = "avocat"
b.goto "https://www.google.fr/search?q="+"#{job}"+" "+"#{ville}" "&dcr=0&ei=1-quWY6BDouYgAb0tZq4CA&start="+"#{k}"+"&sa=N&biw=1280&bih=876"
loop do
l0 = b.div(class:"f kv _SWb", index:0).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l0}')");
l1 = b.div(class:"f kv _SWb", index:1).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l1}')");
l2 = b.div(class:"f kv _SWb", index:2).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l2}')");
l3 = b.div(class:"f kv _SWb", index:3).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l3}')");
l4 = b.div(class:"f kv _SWb", index:4).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l4}')");
l5 = b.div(class:"f kv _SWb", index:5).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l5}')");
l6 = b.div(class:"f kv _SWb", index:6).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l6}')");
l7 = b.div(class:"f kv _SWb", index:7).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l7}')");
l8 = b.div(class:"f kv _SWb", index:8).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l8}')");
l9 = b.div(class:"f kv _SWb", index:9).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l9}')");
k += 10
b.goto "https://www.google.fr/search?q="+"#{job}"+" "+"#{ville}" "&dcr=0&ei=1-quWY6BDouYgAb0tZq4CA&start="+"#{k}"+"&sa=N&biw=1280&bih=876"
break if k == 200
end
end
Launch(b)
As you can see it's very messy, but I'm aware of that and this will be fixed as soon as I get it to work. So when I run it, that's the error I get:
test.rb:17:in `block in Launch': undefined local variable or method `client' for main:Object (NameError)
from test.rb:15:in `loop'
from test.rb:15:in `Launch'
from test.rb:62:in `<main>'
I have searched pretty much everywhere and haven't been able to find a solution to my problem, which is why I created an account here in order to not only look at the questions at others, but post my own.
Thank you everyone
client
is out of scope Launch method.Make it global
@client
or assign as second attribute inLaunch(b, launch)