Web parser refactoring

[nsarrazin]

Describe your feature request

The goal is to move HuggingChat web parsing to a Space that has auto-scaling for increased security.

We should make the parser configurable with a PARSER_CONFIG env var, with at least three options for now:

• SIMPLE (just html fetch)
• PLAYWRIGHT (current parser)
• EXTERNAL (see below)

The external parser should be an external webserver we could host on Spaces for HuggingChat. This would bring multiple benefits:

• We can rely on the security of the existing Spaces infrastructure which is more battle tested
• Decouples parsing load from chat load
• Make the parsing servers more easily scaleable and fault-tolerant so if (when) playwright starts leaking memory we can easily refresh the parser pods without having to constantly kill the nodejs ones
• We can make use of existing python RAG libraries for the parser servers instead of relying on a custom implementation

We should also add a build arg to the docker image INSTALL_PLAYWRIGHT since the simple & external parsers do not need to bundle playwright in the final image.