souvenir
Tag cloud
Picture wall
Daily
RSS Feed
  • RSS Feed
  • Daily Feed
Filters

Links per page

  • 20 links
  • 50 links
  • 100 links

Filters

Untagged links
8 results tagged scraper  ✕   ✕
ScrapeGraphAI https://github.com/ScrapeGraphAI/Scrapegraph-ai
10/12/2024 cluster icon
  • Scrapy : Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be...
  • TextBlob : TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) ...
  • docopt : Command-line interface description language. docopt helps you define interface for your command-line app, and automatically generate parser for it.
  • notifiers : A Python one stop shop for all notification providers with a unified and simple interface.
  • Kartograph : The Kartograph map generator has just one method, which is there to generate SVG maps (surprise).

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).

python scraper library llm
Colly http://go-colly.org/
24/05/2021 cluster icon
  • Go kit : Go is a great general-purpose language, but microservices require a certain amount of specialized support. RPC safety, system observability, infrastru...
  • Revel : A high-productivity web framework for the Go language.
  • Beego : An open source framework to build and develop your applications in the Go way
  • Ponzu : Ponzu is a powerful and efficient open-source HTTP server framework and CMS. It provides automatic, free, and secure HTTP/2 over TLS (certificates obt...
  • Fiber : Fiber is an Express inspired web framework build on top of Fasthttp, the fastest HTTP engine for Go. Designed to ease things up for fast development w...
thumbnail

Colly is a Go framework that provides a clean interface to write any kind of crawler/scraper/spider

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

go scraper crawler framework datamining archive
Scrapoxy https://scrapoxy.io/
23/06/2020 cluster icon
  • Cool QL Cool : CoolQLCool (CQC) is an open source GraphQL server that allows you to turn websites into GraphQL APIs.
  • Envoy Proxy : Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures.
  • Portia : Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web page to id...
  • Audacity : Audacity est un logiciel libre, gratuit et multi-plateforme pour l'enregistrement et l'édition de sons.
  • jina : Jina is a neural search framework that allows to build deep learning search applications in minutes. It provides scalable indexing, querying, understa...
thumbnail

Scrapoxy hides your webscraper behind a cloud.

It starts a pool of proxies to relay your requests.

Now, you can crawl without thinking about blacklisting!

opensource scraper proxy
Cool QL Cool https://coolql.cool/
30/12/2018 cluster icon
  • WireMock : Free and Open Source tool for building mock APIs. Create stable development environments, isolate yourself from flakey 3rd parties and simulate APIs t...
  • Prisma : Prisma turns your database into a powerful GraphQL API which you can either use to build your own GraphQL server or connect to it directly from the fr...
  • Matrix : Matrix is an open standard for decentralised communication, providing simple HTTP APIs and open source reference implementations for securely distribu...
  • GraphQL : GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandabl...
  • Postal : Postal is a complete and fully featured mail server for use by websites & web servers. Think Sendgrid, Mailgun or Postmark but open source and ready f...

CoolQLCool (CQC) is an open source GraphQL server that allows you to turn websites into GraphQL APIs.

graphql scraper opensource server api
Portia https://github.com/scrapinghub/portia
21/02/2016 cluster icon
  • Ossia : A free, open-source, cross-platform intermedia sequencer for precise and flexible scripting of interactive scenarios. Control and score any OSC-compli...
  • Infection Monkey : The Infection Monkey is an open source security tool for testing a data center's resiliency to perimeter breaches and internal server infection. The M...
  • mob : Smooth git handover for remote pair/mob programming. mob is an open source command line tool written in go mob is the fastest way to hand over code vi...
  • Lighthouse : Lighthouse is an open-source, automated tool for improving the quality of web pages. You can run it against any web page, public or requiring authenti...
  • Beekeeper Studio : Open Source cross-platform SQL Editor and Database Manager. Use Beekeeper Studio to query and manage your relational databases, like MySQL, Postgres, ...
thumbnail

Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web page to identify the data you wish to extract, and Portia will understand based on these annotations how to scrape data from similar pages.

scraper opensource tool
Upton https://github.com/propublica/upton
04/02/2014 cluster icon
  • ScrapeGraphAI : ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML...
  • Scrapy : Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be...
  • hunt : Library to detect when DOM elements become visible and disappear on scroll.
  • clipboard.js : Copy text to the clipboard shouldn't be hard. It shouldn't require dozens of steps to configure or hundreds of KBs to load. But most of all, it should...
  • meSing.js : meSing.js is a JavaScript singing synthesis library that uses the Web Audio API's DSP capabilities in conjunction with the meSpeak.js speech synthesis...
thumbnail

Upton is a framework for easy web-scraping with a useful debug mode that doesn't hammer your target's servers. It does the repetitive parts of writing scrapers, so you only have to write the unique parts for each site.

scraper ruby library
Scrapy http://scrapy.org
27/09/2013 cluster icon
  • ScrapeGraphAI : ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML...
  • Goutte : Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extract data from the HTML/XML response...
  • scikit-learn : Machine Learning in Python Simple and efficient tools for data mining and data analysis Accessible to everybody, and reusable in various contexts Buil...
  • pyo : pyo is a Python module written in C to help digital signal processing script creation.
  • Kartograph : The Kartograph map generator has just one method, which is there to generate SVG maps (surprise).

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

python scraper crawler library
Goutte https://github.com/fabpot/Goutte
23/11/2011 cluster icon
  • Its Continuous! : The tweet-sized continuous integration app for PHP development.
  • PHP Quick Profiler : A tool for quickly profiling variables and resources used by PHP.
  • Selenium Webdriver bindings for PHP : This PHP library allows creating functional webdriver tests with PHP.
  • Chromeless : Chrome automation made simple. Runs locally or headless on AWS Lambda. Chromeless can be used to... Run 1000s of browser integration tests in paralle...
  • PHPMD - PHP Mess Detector : What PHPMD does is: It takes a given PHP source code base and look for several potential problems within that source. These problems can be things lik...
thumbnail

Goutte is a screen scraping and web crawling library for PHP.

Goutte provides a nice API to crawl websites and extract data from the HTML/XML responses.

php crawler test scraper
1640 links
Shaarli - The personal, minimalist, super-fast, database free, bookmarking service by the Shaarli community - Theme by kalvn