Urban75 Home About Offline BrixtonBuzz Contact

Today I have mostly been using ${insert development technology here}

Anyone have a solution for me for this problem I have:

Problem: I need to upload files to a website daily. There's no API. It's a web-based file upload. I want to automate it.

I have managed to script it using RPA, meaning it's now a one-click operation - but this requires a real display to work - doesn't work with a virtual display (at least on all my attempts so far).
Selenium works without a display, but I can't get logged into the system using Selenium as it's behind a Google login, and Google detects Selenium and blocks login.

I want to be able to have this run every hour or whatever, without needing to do it manually. I'm guessing I'll need some sort of virtual desktop solution, but don't know where best to look for this. Any other options? If not, any recommendations for virtual desktop providers?
 
Fez909 said:
Anyone have a solution for me for this problem I have:

Problem: I need to upload files to a website daily. There's no API. It's a web-based file upload. I want to automate it.

I have managed to script it using RPA, meaning it's now a one-click operation - but this requires a real display to work - doesn't work with a virtual display (at least on all my attempts so far).
Selenium works without a display, but I can't get logged into the system using Selenium as it's behind a Google login, and Google detects Selenium and blocks login.

I want to be able to have this run every hour or whatever, without needing to do it manually. I'm guessing I'll need some sort of virtual desktop solution, but don't know where best to look for this. Any other options? If not, any recommendations for virtual desktop providers?
Click to expand...
So you need a Google login in order to do the upload?
 
Difficult to say precisely without knowing more details, but for similar operations I have used the Perl module WWW::Mechanize with success in the past.

Possibly not much help if you don’t work with Perl, but you don’t say what languages you want to use.
 
Fez909 said:
Anyone have a solution for me for this problem I have:

Problem: I need to upload files to a website daily. There's no API. It's a web-based file upload. I want to automate it.

I have managed to script it using RPA, meaning it's now a one-click operation - but this requires a real display to work - doesn't work with a virtual display (at least on all my attempts so far).
Selenium works without a display, but I can't get logged into the system using Selenium as it's behind a Google login, and Google detects Selenium and blocks login.

I want to be able to have this run every hour or whatever, without needing to do it manually. I'm guessing I'll need some sort of virtual desktop solution, but don't know where best to look for this. Any other options? If not, any recommendations for virtual desktop providers?
Click to expand...
The only thing I can think of is that you can run a headless Unix machine, so, while there is hypothetically a "display", you don't actually need a real one. But I don't know if your RPA system of choice is OS-dependent.
 
existentialist said:
If Fez909 is uploading files behind a Google login, I fear that cURL isn't quite going to cut the mustard - there's going to be a lot of preamble and token-issuing going on before he even gets to the upload bit.
Click to expand...
It's certainly the other end of the spectrum from an RPA but multiplatform, cron-able, well documented - thought it was worth a shout.
 
de_dog said:
It's certainly the other end of the spectrum from an RPA but multiplatform, cron-able, well documented - thought it was worth a shout.
Click to expand...
Oh, I agree. But my (quite vast) experience in the whole IT field is that the technical validity of a solution always takes a subordinate place to things like "what's the existing infrastructure?". Most places, not entirely unreasonably, won't, for example, sanction the deployment of a Linux box to do some core business function if they're a Windows shop.

And that may well extend to "unsanctioned" utilities like cURL, amazing as it is.
 
existentialist said:
Oh, I agree. But my (quite vast) experience in the whole IT field is that the technical validity of a solution always takes a subordinate place to things like "what's the existing infrastructure?". Most places, not entirely unreasonably, won't, for example, sanction the deployment of a Linux box to do some core business function if they're a Windows shop.

And that may well extend to "unsanctioned" utilities like cURL, amazing as it is.
Click to expand...
sure, I've experienced exactly that in recent times myself. Let's presume Windows only.

I thought cURL with [choice of scripting language] is doable on the local workstation and the hourly requirement suggests the availability of a server or similar always-on device.
 
magneze said:
So you need a Google login in order to do the upload?
Click to expand...
Yes, I login with my Google credentials, then have to do some setup (filling out forms, navigating through pages), then upload the file.
de_dog said:
not a solution but I would start with cURL


Upload files with CURL

CURL is a great tool for making requests to servers; especially, I feel it is great to use for testing APIs.
medium.com medium.com
Click to expand...
It's all completely automated currently. I literally type a single word command at the terminal and my (Python) script opens a browser, navigates to the service, logs in using my Google credentials, does all the page traversing etc, then pulls the details for the form filling from a Trello card that we use for config, and pulls the video file from an FTP, and uploads it to the site. There's no API for this service, which is why I use RPA. I have tried several times with Selenium automation, as I know that works in a headless environment, but unfortunately Google's login page won't allow Selenium past it - it detects that "The browser is being controlled by automation software" and blocks login.

It used to be that you could set something in your Google account to allow 'unsafe' logins but they've put a stop to that.

For whatever reason, RPA is able to bypass this and gets me logged in and does everything I need. But it needs to be triggered manually by me and I have a browser pop up and it does its thing and then closes.

I want to be able to run this on my server (Linux/Ubuntu), but the lack of display means RPA fails.

edit: by RPA I mean RPA-python (GitHub - tebelorg/RPA-Python: Python package for doing RPA)
 
prunus said:
Difficult to say precisely without knowing more details, but for similar operations I have used the Perl module WWW::Mechanize with success in the past.

Possibly not much help if you don’t work with Perl, but you don’t say what languages you want to use.
Click to expand...
Thanks, this looks interesting, but don't really want to have to learn Perl to get this done. If I works, though, it'll be worth it...

Hopefully someone else has an alternative for me before I go down that road :)
 
Fez909 said:
Thanks, this looks interesting, but don't really want to have to learn Perl to get this done. If I works, though, it'll be worth it...

Hopefully someone else has an alternative for me before I go down that road :)
Click to expand...

I think it will [work for you] - although I’ve said “in the past” it’d be more accurate to say I wrote a number of (pretty mission-critical) scripts in the past that are still running and automating logins to google and similar services, uploading files via authenticated web forms (and doing other things) probably 100 times a day between them.

If you can program Python knocking up a perl script won’t drive you more than 1/4 insane :-D (I actually prefer perl myself but understand I’m in a minority). The module documentation and examples on Cpan will walk you through most of it :)

Good luck with your search!
 
If you have physical access to your server then a HDMI dummy plug might help get round part of the issue.
 
Fez909 said:
Yes, I login with my Google credentials, then have to do some setup (filling out forms, navigating through pages), then upload the file.

It's all completely automated currently. I literally type a single word command at the terminal and my (Python) script opens a browser, navigates to the service, logs in using my Google credentials, does all the page traversing etc, then pulls the details for the form filling from a Trello card that we use for config, and pulls the video file from an FTP, and uploads it to the site. There's no API for this service, which is why I use RPA. I have tried several times with Selenium automation, as I know that works in a headless environment, but unfortunately Google's login page won't allow Selenium past it - it detects that "The browser is being controlled by automation software" and blocks login.

It used to be that you could set something in your Google account to allow 'unsafe' logins but they've put a stop to that.

For whatever reason, RPA is able to bypass this and gets me logged in and does everything I need. But it needs to be triggered manually by me and I have a browser pop up and it does its thing and then closes.

I want to be able to run this on my server (Linux/Ubuntu), but the lack of display means RPA fails.

edit: by RPA I mean RPA-python (GitHub - tebelorg/RPA-Python: Python package for doing RPA)
Click to expand...
This may help -

stackoverflow.com

Chrome is being controlled by automated test software

I am running automated tests in Chrome with Serenity BDD (Selenium). I had to download a new ChromeDriver, because my tests could not run -> The test would open ChromeDriver but could not "Browse as
stackoverflow.com stackoverflow.com
 
if you can get authed by Google without the browser window (receive cookies / token), could you then start a [automated] authenticated browser session with the newly accquired credentials? pycURL?
dunno, I haven't looked it up ;) I suppose the suggestion is that if you have to work around the auth then is there a better way to impersonate a browser. glhf
 
Fez909 said:
Yes, I login with my Google credentials, then have to do some setup (filling out forms, navigating through pages), then upload the file.

It's all completely automated currently. I literally type a single word command at the terminal and my (Python) script opens a browser, navigates to the service, logs in using my Google credentials, does all the page traversing etc, then pulls the details for the form filling from a Trello card that we use for config, and pulls the video file from an FTP, and uploads it to the site. There's no API for this service, which is why I use RPA. I have tried several times with Selenium automation, as I know that works in a headless environment, but unfortunately Google's login page won't allow Selenium past it - it detects that "The browser is being controlled by automation software" and blocks login.

It used to be that you could set something in your Google account to allow 'unsafe' logins but they've put a stop to that.

For whatever reason, RPA is able to bypass this and gets me logged in and does everything I need. But it needs to be triggered manually by me and I have a browser pop up and it does its thing and then closes.

I want to be able to run this on my server (Linux/Ubuntu), but the lack of display means RPA fails.

edit: by RPA I mean RPA-python (GitHub - tebelorg/RPA-Python: Python package for doing RPA)
Click to expand...
You might be able to automate the login via OAuth if that's the main issue.
 
I have been specifically not using vba, as it is not allowed, too many issues with having to enable macros across the service. Hired to basically create a fee of works estimation sheet that a very large department will be using for huge works using a lot of predetermined values for employees, a lot of predetermined values for equipment and also needs a free entry section which needs to be both locked down and somehow completely customisable. Excel guy there is great but wow is he behind some technological changes, pivots seem a mystery but complicated vba and ridiculously long formulas and lots of fixed references to cells on other sheets sure.

Have managed to cobble something together which basically does everything now but locking it down is proving a right pain in the ass, I have everything properly setup with table references (instead of cell references that mean nothing). Everything locked when I run the macro to do so, conditional changes all over the place when they select the modifier they insisted upon (which basically changes the entire lot to another set of values and is only a maybe as to it being used at all...). Huge error messages if people manage to somehow do something ridiculous that they should not be able to anyway but well I have helped people with computers before.

Thing I am having the main issue with it PQ forming the quote boxes, its drawing from 15 main query's that are appended depending on variables, to then output to two others that are filtered based on various factors. It all works fine and auto sizes but without vba I can't make it update when a change is made. Trialled 5 minute query refresh which works fine, ish. I don't like having to add a message saying well it may say x but well manually refresh all this shit because you can't trust it if you made an entry since the last refresh. Got refresh on load on which could be another route, save work before reopening file to get your results but its messy.

Probably didn't help I used a legacy thing as a template (literally never done this type of work so some things they implemented were useful as a basic guide), tho my filesize is now 10% what theirs was and we have CAD quality laptops from this year and I rewrote absolutely everything so no legacy stuff remains.... could be an issue but its got proper naming conventions for tables etc so should be fine.

I could add 1 minute refreshes but that feels like it will be annoying to see all the time, basically they just want to do some data entry and have it spit out the results of costs, manpower, contract length, bla bla bla and see there are no errors. Spent all day trying to break it in different ways and got nowhere, is there some way of making sure its watertight? Also a little baffled since they gave me months for this and I have been 90% done (ha) since week 2 of 4 so far and well I like being paid and actually the work but its more than 2 months til contract end.

Looking at similar stuff but they want BI/SQL, is there some way of getting this experience/certs of whatever an actual hiring manager will look at without saying thats irrelevant outside a work environment? Otherwise I have to go back to my last industry which pays well but thats kind of cos I already know things, I tend to want to keep learning things. Not just sit as the font of all knowledge about something I know, well workwise anyway, otherwise I love being the expert, until that gets boring to deal with lol.
 
iamwithnail said:
Kinda related, but today i'm nervously holding my breath a bit as we scaled up to a bajillion servers/pods and launched a new product to hundreds of thousands of people.
Click to expand...
I’m still doing this but at a different place and bigger scale. Terraform to build kube clusters on AWS (well - to build everything tbf), and argocd to manage deployments in to the clusters. Still bloody Jenkins as a wrapper for the CI around it tho.
 
Anyone using ChatGPT? I find it's great as a replacement for Googling errors/Stack Overflow/looking up documentation.

It's not always right, and sometimes it does Weird Things, but with a few messages back and forth, it's helping me save quite a bit of time.

I also have it review my code - it's great for refactoring, but it often just says, "add more test and error handling" as if I didn't know I'm not doing that enough :facepalm:

Yesterday I was doing something that meant scanning through videos on a video player on a website and copying and paste the seek time to another sheet. Very tedious, and made worse because the timestamps had to be in hh:mm:ss.000 format.

I asked ChatGPT to write a Userscript for me, and it did it in a few minutes. It added a copy that copied the <video> element's currentTime to the clipboard in the format I wanted.

It probably cost me more time than I saved, but the annoyance factor was reduced significantly.

I liked how it put itself as the author in the Userscript manifest/declaration :D

Code: 
// ==UserScript==
// @name         Video Time Copier for #### with Polling
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  Try to copy the video player's current seek time to the clipboard in hh:mm:ss.sss format when the video is played.
// @author       ChatGPT
// @match        https://####/feature/*
// @grant        none
// ==/UserScript==
 
You must log in or register to reply here.

Similar threads

A
Have I Been Pwned?
2
Replies
50
Views
2K
Chz
Chz
Boudicca
Using technology for the elderly to call for help
2 3
Replies
72
Views
2K
Boudicca
Boudicca
PTK
Decline in Telephone Quality and Other Regressions in Technology
Replies
6
Views
217
Puddy_Tat
Puddy_Tat
Ranbay
Artificial Intelligence Developments (CatGPT etc)
Replies
4
Views
455
Rob Ray
Rob Ray
RiderOnTheStorm
Help Radio unlock code needed, pls
Replies
12
Views
447
Saul Goodman
Saul Goodman
Back
Top Bottom