Catching the transient cookies: Log in, then crawl
The old way Sometimes all you need is a quick crawl within a site, which requires to log in first. There are two main techniques I can think about: One is to POST the login form with your script, and get the necessary cookie setting. The second is to login manually with a browser, and [...]