So I've been working on some scraping scripts with Ruby, using the gem watir lately. I used to put the results in a .json file but I recently started working with MySQL2. So here's the deal:
I have created a db called scrap, with a table called links, which has 3 columns: id, job, link. The script I'm working on for this is very simple: it makes a query on google search, takes the URLs of all the links, then goes to the next page. So here's the boy:
require "watir"
require "json"
require "mysql2"
b = Watir::Browser.new(:firefox)
client = Mysql2::Client.new(:host => "localhost", :username => "root", :password => "xxx", :database => "scrap", :encoding => "utf8")
def Launch(b)
job = "cabinet avocat"
ville = "paris"
k = 0
j = "avocat"
b.goto "https://www.google.fr/search?q="+"#{job}"+" "+"#{ville}" "&dcr=0&ei=1-quWY6BDouYgAb0tZq4CA&start="+"#{k}"+"&sa=N&biw=1280&bih=876"
loop do
l0 = b.div(class:"f kv _SWb", index:0).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l0}')");
l1 = b.div(class:"f kv _SWb", index:1).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l1}')");
l2 = b.div(class:"f kv _SWb", index:2).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l2}')");
l3 = b.div(class:"f kv _SWb", index:3).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l3}')");
l4 = b.div(class:"f kv _SWb", index:4).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l4}')");
l5 = b.div(class:"f kv _SWb", index:5).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l5}')");
l6 = b.div(class:"f kv _SWb", index:6).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l6}')");
l7 = b.div(class:"f kv _SWb", index:7).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l7}')");
l8 = b.div(class:"f kv _SWb", index:8).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l8}')");
l9 = b.div(class:"f kv _SWb", index:9).text
client.query("INSERT INTO links(job, link)
VALUES('#{j}, '#{l9}')");
k += 10
b.goto "https://www.google.fr/search?q="+"#{job}"+" "+"#{ville}" "&dcr=0&ei=1-quWY6BDouYgAb0tZq4CA&start="+"#{k}"+"&sa=N&biw=1280&bih=876"
break if k == 200
end
end
Launch(b)
As you can see it's very messy, but I'm aware of that and this will be fixed as soon as I get it to work. So when I run it, that's the error I get:
test.rb:17:in `block in Launch': undefined local variable or method `client' for main:Object (NameError)
from test.rb:15:in `loop'
from test.rb:15:in `Launch'
from test.rb:62:in `<main>'
I have searched pretty much everywhere and haven't been able to find a solution to my problem, which is why I created an account here in order to not only look at the questions at others, but post my own.
Thank you everyone
clientis out of scope Launch method.Make it global
@clientor assign as second attribute inLaunch(b, launch)