Some tips PySpark and Pandas in Databricks

We have a list: ListA = ['a', 'b', 'c', 'd', 'e'] And we want to send it by email to someone. So what we have to do, first, is save this list as txt file in a temporary location in databricks, and then we open it and put it in a variable, so we can be able to send it by email. Like the example: import boto3 import smtplib import matplotlib.pyplot as plt from email import encoders from email.mime.text import MIMEText from email.mime.image import MIMEImage from email.mime.multipart import MIMEMultipart from email.mime.application import MIMEApplication server = smtplib.SMTP('smtp-mail.outlook.com:587') #i am using Outlook server.ehlo() server.starttls() server.login('email@outlook.com.br', 'password@97') #Here is your id and password sender = "email@outlook.com.br" #here is your id again recipient = ['destiny@outlook.com.br'] #here the email id from who gonn...

To round minute to nearest hour you have to use: date_trunc('hour', 'YOUR DATE_TIME' + interval '30 minute') DATE_TRUNC function rounds to the hour that appears in the string, so if we add +30 minutes when you have a HH:30+, this should round to next hour and when you have HH:-30 this will round to the hour that appears in your string. So if hour is: '2022-11-21 00:05:00', the new time will be: '2022-11-21 00:00:00' But if hour is: '2022-11-21 00:55:00', the new time will be: '2022-11-21 01:00:00' Between 0 and 30 minutes: Rounds to the same hour in your date_time Between 31 and 59 minutes: Rounds to the next hour based on your date_time

Use something like this: db.getCollection("YOURTABLE").find( { aggIndex : /^FIRST_INDEX \| SECOND_INDEX.*/i } ); When we have a pipe character: | and we try to find() something with this pipe, the mongoDB shell understand it's like an OR statement. So we just have to add "\" before the pipe |. For LIKE I used /^TEXT.*/i and it works great! Using LIKE (SQL): aggIndex LIKE 'FIRST_INDEX|SECOND_INDEX%'; LIKE in MongoDB: aggIndex : /^FIRST_INDEX \| SECOND_INDEX.*/i The /^ determines the beginning. The .*/i determines that we shall have more characters after this point, independently if they are lower or upper case. It works for deleteMany() too.

Hello, i'm creating this blog with some tips that i took a long time to find to some specific problems How do I update a field value from some column searching by id from a column in Pandas? This is similar to a query in SQL like: UPDATE tableX SET fieldX = 'valueX' WHERE ID = 'idX' So: yourDF.loc[yourDF['ColumnNameWhereIsYourId'].isin( [HERE YOU CAN PUT A LIST OF ID's or just one like '67ae47e2-f04d-415e-a2b5-2c212388044a']), 'NameOfColumnThatYouHaveToUpdateTheValue'] = 'Value you want' Example: I have a dataframe pandas like this: I wanna change name jhon from id 67ae47e2-f04d-415e-a2b5-2c212388044a to Jonathan. So i will do this: df.loc[df['column_id'].isin( ['67ae47e2-f04d-415e-a2b5-2c212388044a']), 'ColumnName'] = 'Jonathan' My dataframe now: This will update a value from a column in the row that your ID is in.