The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
Problems getting text from web pages
Contributor I
Hi,
I have an Excel file with 330+ links to web pages. I need to extract the text from all the web pages to do a clustering task.
I'm not able to achieve this with the usual process:
get pages >> data to documents >> loop collection > extract content >> documents to data.
The problem is that the operators are only able to extract (for all the pages) the same that we get with the "view page source" directly in the browser. So, what I get is an empty Text attribute.
I tested with only one link (https://dre.pt/dre/detalhe/despacho/3219-2020-130112149) with the operator Get Page. This is what I get in the extracted document:
How can I get the text from this? Can you help me, please?
I have an Excel file with 330+ links to web pages. I need to extract the text from all the web pages to do a clustering task.
I'm not able to achieve this with the usual process:
get pages >> data to documents >> loop collection > extract content >> documents to data.
The problem is that the operators are only able to extract (for all the pages) the same that we get with the "view page source" directly in the browser. So, what I get is an empty Text attribute.
I tested with only one link (https://dre.pt/dre/detalhe/despacho/3219-2020-130112149) with the operator Get Page. This is what I get in the extracted document:
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="format-detection" content="telephone=no" />
<script type='text/javascript'>window.OutSystemsApp = { basePath: '/dre/' };</script>
<meta http-equiv="Content-Security-Policy" content="base-uri 'self'; child-src * gap:; frame-src * gap:; connect-src *; default-src 'self' 'unsafe-inline' *.google-analytics.com *.hotjar.com *.googletagmanager.com *.dre.pt *.hotjar.io *.doubleclick.net *.knightlab.com *.google.com *.google.pt gap: 'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; img-src * blob:; script-src 'unsafe-inline' * 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-ancestors *.incm.pt *.dre.pt 'self' gap:; report-uri /SecurityUtils/rest/Report/ReportViolations?Params=RyAoRWX6RljInm%2B3hwnwrmeQNc96mBOvSkaT58%2FC4zhhZv0xIQAa3h3ft2scL67pequ212Wx6csuqpGp8%2B%2F%2B%2Bw%3D%3D; " />
<meta http-equiv="X-Content-Security-Policy" content="base-uri 'self'; child-src * gap:; frame-src * gap:; connect-src *; default-src 'self' 'unsafe-inline' *.google-analytics.com *.hotjar.com *.googletagmanager.com *.dre.pt *.hotjar.io *.doubleclick.net *.knightlab.com *.google.com *.google.pt gap: 'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; img-src * blob:; script-src 'unsafe-inline' * 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-ancestors *.incm.pt *.dre.pt 'self' gap:; report-uri /SecurityUtils/rest/Report/ReportViolations?Params=RyAoRWX6RljInm%2B3hwnwrmeQNc96mBOvSkaT58%2FC4zhhZv0xIQAa3h3ft2scL67pequ212Wx6csuqpGp8%2B%2F%2B%2Bw%3D%3D; " />
<meta http-equiv="X-WebKit-CSP" content="base-uri 'self'; child-src * gap:; frame-src * gap:; connect-src *; default-src 'self' 'unsafe-inline' *.google-analytics.com *.hotjar.com *.googletagmanager.com *.dre.pt *.hotjar.io *.doubleclick.net *.knightlab.com *.google.com *.google.pt gap: 'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; img-src * blob:; script-src 'unsafe-inline' * 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-ancestors *.incm.pt *.dre.pt 'self' gap:; report-uri /SecurityUtils/rest/Report/ReportViolations?Params=RyAoRWX6RljInm%2B3hwnwrmeQNc96mBOvSkaT58%2FC4zhhZv0xIQAa3h3ft2scL67pequ212Wx6csuqpGp8%2B%2F%2B%2Bw%3D%3D; " />
<meta name="viewport" content="viewport-fit=cover, width=device-width, initial-scale=1" />
<script type="text/javascript">
(function () {
function appendMetaTagAttributes(metaTag, attribute, values) {
var elem = document.querySelector("meta[name=" + metaTag + "]");
if (elem) {
var attrContent = elem.getAttribute(attribute);
elem.setAttribute(attribute, (attrContent ? attrContent + "," : "") + values.join(","));
}
}
if (navigator && /OutSystemsApp/i.test(navigator.userAgent)) {
// If this app is running on the native shell, we want to disable the zoom
appendMetaTagAttributes("viewport", "content", ["user-scalable=no", "minimum-scale=1.0"]);
}
})();</script>
<script type="text/javascript" src="/dre/scripts/OutSystemsManifestLoader.js?3F3fZzzNKkqKoP2DsjtxFw"></script>
<script type="text/javascript" src="/dre/scripts/OutSystems.js?RnlDcii3Xz75iIHHERIZtA"></script>
<script type="text/javascript" src="/dre/scripts/OutSystemsReactView.js?0bmp5RZ49TZneVNXnO6ymw"></script>
<script type="text/javascript" src="/dre/scripts/cordova.js?7KqI9_oL9hClomz1RdzTqg"></script>
<script type="text/javascript" src="/dre/scripts/NullDebugger.js?pG_2wlzY3NYiuKZRtoLyQQ"></script>
<script type="text/javascript" src="/dre/scripts/DRE.appDefinition.js?otw_Nv9Nr+Q7EbWK92qVcw"></script>
<script type="text/javascript" src="/dre/scripts/OutSystemsReactWidgets.js?IdWooa_erXOfwU01FQUTuA"></script>
<link type="text/css" rel="stylesheet" href="/dre/css/_Basic.css?EqGzAe81QbZLXJyfY3oLwA"></link>
<script type="text/javascript">OSManifestLoader.indexVersionToken = "8Ah8C5iCZm4zS2Ya5zFMJg";
</script>
</head>
<body>
<div id="reactContainer"></div>
<noscript><span>JavaScript is required</span></noscript>
<script type="text/javascript" src="/dre/scripts/DRE.index.js?IcuQoXtODBlF5z87QVycVQ"></script>
</body>
</html> How can I get the text from this? Can you help me, please?
0
Answers
The webpage you are trying to crawl is java generated and has a lot of endpoints that provide the data that you are trying to grab.
I created a template that you can use an adjust to get the text you are tying to obtain.
I put the process running, and it resulted for one link (even do I cannot find where the link is defined!).
I need now to make the process run for several other links. Nevertheless, I was not able to address your comment:
"Adjust Line 1001 with a macro to loop through the documents you might need. You may also need to adjust the request header token if it expires."
because I don´t know to use macros, and do not understand how to request the header token.
Can you help me, please?
Thanks.