still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
[resource]+[mountain/stone/field/valley/tree] in German is a very common pattern for Jewish surnames. A quirk of how surnames were adopted in German-speaking regions in the 18th century
Sugar mountain at least makes sense, unlike e.g. wine stone
I read somewhere once that Sam Altman sounds like a name for a fictional villain made up by Hideo Kojima for Metal Gear Solid, and it will never not make sense.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
How would we reformulate it today?
Make of that what you will.
Sugar mountain at least makes sense, unlike e.g. wine stone
News at never.